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Preface 



For the thirty first year, the Research and Theory Division of the Association for Educational 
Communications and Technology (AECT) is sponsoring the publication of these Proceedings. Papers 
published in this volume were presented at the annual AECT Convention in Orlando, EL. A limited 
quantity of these Proceedings were printed and sold in both hardcopy and electronic versions. This year’s 
Proceedings has two sections - Section 1 includes research and development papers and Section 2 includes 
papers on the practice of educational communications and technology. These Proceedings are submitted to 
the Educational Resources Clearinghouse (ERIC) System for indexing. Proceedings volumes are available 
to members at AECT.ORG. 

REEEREEING PROCESS; Papers selected for presentation at the AECT Convention and included in these 
Proceedings were subjected to a reviewing process. All references to authorship were removed from 
proposals before they were submitted to referees for review. Approximately sixty percent of the 
manuscripts submitted for consideration were selected for presentation at the convention and for 
publication in these Proceedings. Additionally, Proceedings editors review papers prior to their publication. 
The papers contained in this document represent some of the most current thinking in educational 
communications and technology. 
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Editor 
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Effects of Instructional Setting and Interaction Cues in Collaborative 

Computer-Based Instruction 

Christy M. Alarcon 
Arizona State University 

Jeremy I. Tutty 
Boise State University 

James D. Klein 

Arizona State University Abstraet 

The purpose of this study was to investigate the effects of instructional setting and interaction cues on 
performance, attitudes, interaction behaviors and time on task during collaborative, computer-based instruction 
(CBI). Four sections of a computer literacy course for pre-service teachers were assigned to either a face-to-face or 
an online collaborative treatment condition. Mixed-ability dyads were formed and then randomly assigned to receive 
either scripted or non-scripted cues. Results for performance measures did not reveal any significant differences 
between treatment conditions. Findings for attitude indicated that students in the face-to-face instructional setting 
responded more positively toward the computer program and collaboration method than those in the online 
instructional setting. Results for interaction behavior revealed that face-to-face dyads exhibited significantly more 
on-task behavior, while online dyads exhibited significantly more off-task behavior. Furthermore, students who 
received non-scripted cues exhibited significantly more off-task behavior than those who received scripted cues. 
Time data revealed that dyads in the online collaborative condition spent significantly more time on orientation 
screens than dyads in the face-to-face condition. Implications for the design and development of computer-based 
learning environments that involve collaborative strategies are provided. 

Introduction 

Findings from recent research provides support for the use of collaborative learning strategies in online 
settings (Blocher, 2005; Brewer, 2004; Brewer & Klein, 2006; Brewer, Klein, & Mann, 2003; Ku, Lohr, & Cheng, 
2004; Mackie & Gutierrez, 2005; Tutty & Klein, 2008). The effectiveness of computer-mediated, collaborative 
learning may rely upon the amount and quality of interaction among members of a group. According to Johnson and 
Johnson (2004), the nature of collaborative learning encompasses dialogues generated by interactions with other 
students. Sherman (1994) further elaborates that, “Cooperative learning studies in which group member interactions 
have been recorded and analyzed indicate that achievement and attitude differences are related to the type and 
amount of verbal interaction between students within cooperative groups”(p 6). 

Even though some research has been done to demonstrate the positive effects of incorporating collaborative 
strategies in online environments (Brewer, 2004; Brewer & Klein, 2006; Tutty & Klein, 2008; Uribe et ah, 2003), 
there appears to be insufficient empirical support applying the results of existing research to the design and 
development of today’s online collaborative learning environments (Reeves et ah, 2004; Skiba, 2006). Criticism of 
these studies indicates an emphasis on the quantitative nature of the online environment rather than focusing on the 
quality of learning and interaction behaviors. 

A strategy to facilitate interaction between group members, and potentially influence learner achievement, 
is the utilization of embedded prompts. Sherman (1994) developed CBI that included verbal interaction cues that 
facilitated summarizing and explaining between partners at various points in the program. In the cued version, the 
cues assigned students to summarizer or explainer roles. The summarizer was directed by name to recall the 
objectives and summarize the information presented in the lesson. The listener was directed by name to listen 
carefully and ask questions about things that were unclear, omitted, or in error. In the noncued version, there were 
similar opportunities for students to interact, but the program did not explicitly prompt the students to do so. The 
results of this study indicated that students who used the cued version of the program performed significantly better 
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on learning outcomes than students who used the noncued version. In further examining group composition, these 
achievement results were consistent for both higher and lower-ability students. 

The advantages of interaction cues as a systematic collaborative learning strategy were also demonstrated 
in an earlier study conducted by McDonald, Larson, Dansereau and Spurlin (1985). Throughout the instructional 
program, the students in each pair were instructed to alternate between the roles of recaller and listener/facilitator as 
they studied two 2500-word passages and were tested on their individual knowledge of the content. One of the 
students in the dyad, serving as the recaller, attempted to summarize from memory what had been learned. The other 
student, as the listener/facilitator, attempted to correct errors in the recall and facilitated further the organization and 
storage of the material. The results of this study revealed that interaction cues given to students who studied in pairs 
outperformed students who studied alone. These results suggested that the combination of working in pairs and 
incorporating interaction cues as a collaborative learning strategy enhanced individual posttest performance. 

The purpose of the current study was to investigate the effects of instructional setting (face-to face and 
online) and interaction cues (non-scripted cued interactions and scripted cued interactions) and during collaborative 
computer-based instruction. The study addressed the following research questions: 

1 . What are the effects of instructional setting (face-to face versus online) in collaborative computer- 
based instruction on achievement, attitudes, student interaction behaviors, and time on task? 

2. What are the effects of interaction cues (non-scripted versus scripted) in collaborative computer- 
based instruction on achievement, attitudes, student interaction behaviors, and time on task? 

3. Do interaction cues and instructional setting interact to affect achievement, attitudes, student 
interaction behaviors, and time on task? 

Method 



Participants and Setting 

Participants were 86 undergraduate preservice teachers enrolled in four sections of a required computer 
literacy course at a public university in the northwestern United States. The majority of participants were female 
(67.5%), with an average age of 25, and were from all major content areas. The course covered the use of 
spreadsheets, databases, word processing, and presentation software in the classroom. The course prepared students 
for a state-mandated technology competency exam, which must be passed to receive credit for the course. The 
course met twice a week for 75 minutes per session in a closed computer lab on campus. Collaborative groups were 
often used during activities related to integration of technology in the classroom. 

Materials 

A computer-based instructional (CBI) program, entitled Building Excel Spreadsheets, was developed for 
this study. The CBI presented basic functions of Microsoft Excel 2007 and the application of spreadsheets in the 
classroom. It was based on a program from a previous study (Tutty, 2006); the original content was adapted from the 
book. Curricular computing: Essential skills for teachers (Pollard, VanDehey, & Pollard, 2003). The CBI used in 
this study followed a similar design and lesson sequence as that of Tutty’ s program, with modifications that included 
content covering the latest version of Excel and embedded interaction cues. 

Eour versions of Building Excel Spreadsheets were developed for this study: non-scripted cued/face-to- 
face, scripted cued/face-to-face, non-scripted cued/online, and scripted cued/online. The introductory material, 
instructional content, practice activity and group application project were exactly the same in all four versions. 
However, both scripted cued versions (scripted cued/face-to-face and scripted cued/online) included explicit group 
member interaction cues embedded throughout the instruction, while the non-scripted cued versions (non-scripted 
cued/face-to-face and non-scripted cued/online) excluded these cues. The scripted cued versions included two types 
of group member interaction cues — content summary cues and clarifying cues. 
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The non-scripted cued versions of the program prompted students to review the information with their 
partner before proceeding to the practice activity and group application project. Thus, participants in the non- 
scripted cued treatments received cues to interact, but they were general in nature. The scripted cued versions of the 
program contained interaction cues that directed a student by name and provided specific instructions to interact 
with their partner. Content summary cues directed students to explain to their partner the information that was 
presented in the lesson. Clarifying cues told students to pay close attention to these explanations, ask their partner 
questions and paraphrase what was explained to them. 

The practice activity consisted of 67 screens containing a classroom based scenario. Participants assumed 
the role of a classroom teacher with a task of developing a class gradebook. To successfully complete the activity, 
students were required to contain elements of a gradebook and to be formatted appropriately. In addition, it needed 
to be formatted to calculate student performance in each of the assignment categories, student performance for the 
grading period, each student's final grade, and the highest and lowest grades for each assignment. Students then 
constructed a bar chart to display the data. Students worked together in the dyads to complete a single spreadsheet 
project. 



The group application project consisted of six instructional screens. Each dyad applied the individual skills 
they learned from the practice activity section to create a second spreadsheet based on a problem scenario that 
involved analyzing a student budget for an entire year. No additional instruction was provided, but students were 
given the option to access their previous content track (designer or programmer) for reference through a separate 
browser window. For this activity, participants were required to use their critical thinking skills to determine how 
best to complete the activity. At the end of instruction, each dyad submitted one application project. 

To encourage collaboration throughout the instruction, the practice activity was structured in two parallel 
tracks, following a procedure used in a previous study (Tutty, 2006). The two tracks — designer and programmer — 
each contained different skills required to complete the practice problem. One participant was unable to receive all 
of the information necessary to complete the exercise independently. Once a student selected a specific track by 
clicking on the appropriate button, they were unable to access the other track. This required each student to 
carefully communicate with their partner by teaching and learning their designated skills from one another. 

Both designer and programmer tracks presented participants with a list of concepts and terms that should be 
defined and achieved. Each track contained specific step-by-step instructions, explanations and screen captures of 
Excel 2007 to illustrate the definition and function of each element in the instruction. The CBI presented the 
objectives and content within both tracks to ensure that all participants were aware of both sets of skills required of 
the individual posttest. Each track also contained a reminder that the participants were required to know both sets of 
skills in order to effectively complete the group application project. 

Procedures 

Each of the four sections of the computer literacy course was assigned to one of the instructional setting 
treatment groups used in this study. As a result, two class sections were assigned to the face-to-face condition and 
the other two were assigned to the online one. Within each class, participants were blocked by prior knowledge of 
the course content and randomly assigned to dyads of heterogeneous prior knowledge composition (each dyad 
contained a student with high prior knowledge of computer basics and a student with low prior knowledge). Prior 
knowledge blocking was based upon performance on a general computer proficiency pretest that was administered 
prior to data collection. As a result, there were approximately 12 dyads within each class section and each dyad was 
randomly assigned to either a scripted cues or non-scripted cues treatment. 

For the face-to-face treatments, students in each dyad were physically seated next to each other (one 
student per computer). In the online treatments, students in each dyad were seated at separate computer 
workstations in the classroom. This particular seating arrangement required the online dyads to communicate only 
through the Blackboard chat room. Each dyad was identified by a numerical code, which corresponded with the 
type of cued interaction and instructional setting treatment each participant was assigned to. In the face-to-face 
condition, student name plates were placed on each computer station. Each student was able to identify their 
assigned computer station and assigned partner with these name plates. For dyads in the online condition, the 
researcher preset the dyad assignments in advanced. Participants in the online version automatically communicated 
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with their assigned partners when they logged in to the Blackboard chat room. Each dyad remained at their 
designated computer stations throughout the duration of the treatment period. 

Prior to the study, all participants experienced at least two months of working in formal and informal 
collaborative learning groups in the computer literacy course at the beginning of the semester. These collaborative 
activities were related to the course curriculum and were not part of the study in any way. To ensure that students 
were comfortable with the Blackboard features required in the course, the regular curriculum included a review of 
Blackboard during the first week of the semester. The role of the instructors was minimal and they did not 
participate in providing instruction during the implementation of the study. 

The primary researcher of the study and a research assistant were trained in classroom observation and data 
collection of student interactions within dyads. Training of observation procedures consisted of reviewing literature 
on categorizing interactions and applied practice using the observation sheet. The applied practice involved 
observing both face-to-face and online dyad interactions. For preliminary online observations of student 
interactions, the primary researcher and the research assistant conducted a content analysis of a transcript during a 
brief chat session between two students in a dyad. Using the same observation sheet as the one used for the face-to- 
face observations, the observers indicated the number of instances for each interaction category. 

During the implementation of the study, the primary researcher and the research assistant observed all 
dyads for every class section (approximately 12 dyads per class section). Each instance of collaboration was 
indicated on an observation form. Raters identified each dyad by a numerical code, but were unaware of the dyad’s 
type of cueing treatment to avoid any bias. Each dyad was observed in two-minute intervals at various points 
throughout the program comprising a minimum of 25 minutes of observation during the instruction (adapted from 
Tutty, 2006). The following interaction behaviors were observed during instruction: summarizing, explaining, 
identifying errors, helping behaviors and discussing. These behaviors were grouped under one category: on-task 
behavior. On-task behavior is conceptually defined as the number of times a dyad discussed content related to 
instruction. Non-task related behaviors were categorized as off-task behavior, and included receiving solicited and 
unsolicited help, and verbal encouragement. 

All dyads were given three, 75-minute class periods to complete the instructional program and assessments. 
Observations of student interactions were collected for each dyad during the first two days of treatment. In the first 
class period, the participants worked on the web-based instruction in dyads at their own pace. They were allowed to 
use the entire 75-minute time period. In the second class period, the participants completed the group project. Then, 
in the third class period, they took an online posttest on what they had just learned about Excel and spreadsheets. 
Finally, they were asked to complete an online attitude survey regarding the instruction and their experiences 
working in collaborative dyads. A small sample of students was asked to volunteer for a follow-up interview on their 
experiences during the computer-based instruction. The researcher used a standard interview form and the interview 
with each student took about 5-10 minutes to complete. 

Data Collection Instruments 

There were several measures used in this study — a pretest, a posttest, a group application project, a student 
attitude survey, an interview protocol, observations of student interactions and a measure of time on task. Several 
weeks prior to the implementation of the study, participants completed a general computer proficiency pretest. 
Results were used to assign students to dyads of mixed prior knowledge of basic computer skills. A 25-item, 
multiple-choice, posttest was administered online following the completion of the CBI and measured student 
knowledge of the content covered in the program. The posttest items were based on the objectives of the lesson, 
measuring skills and knowledge covered in the instructional program. During this assessment, participants were 
required to identify concrete or defined concepts related to spreadsheet functions, terms, and formulas. 

An attitude survey measured student interest, motivation, confidence, enjoyment, and attitudes toward 
working with a partner and towards the instruction. The survey contained 15-Likert scale items, ranging from 1 - 
strongly disagree to 5 - strongly agree. Three open-ended questions were also included. Most of the items were 
taken from versions of attitude surveys used in previous studies on collaborative learning and using interaction cues 
(Sherman, 1994; Tutty, 2006). Some of these items had been modified and additional items were included in the 
final attitude survey developed for this study. 
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A sample of 1 1 participants from each of the two instructional setting conditions participated in an 
interview and were asked both forced-response and open-ended questions related to their perspectives and opinions 
of the instructional program, the features of the CBI, and their perception of collaborative learning during 
instruction. A six-item interview protocol adapted from Tutty’s (2006) study was used to collect more in-depth 
attitude data. 

A rubric was used to evaluate group project performance and was adapted from a grading rubric used in a 
previous study (Tutty, 2006). Due to the nature of the group project problem scenario, there was not a single correct 
answer. Students were evaluated based on the inclusion of the required contents, the accuracy of their calculations, 
and the format of their output data and spreadsheet. The maximum possible score on the rubric was 30 points. 

Group project performance represented an achievement measure for both members of a dyad. The principal 
researcher and a research assistant both graded the group projects using the scoring rubric. During a preliminary 
training on how to score the group projects, the graders discussed discrepancies in their scoring to establish 
consistency in rubric categorizations and account for interrater reliability. 

The amount of time spent on the program was recorded by the CBI for each individual. Time on task was 
determined as the amount of time each individual spent on orientation screens, instructional screens, and the practice 
activity. The total elapsed time for the entire program for each participant was also calculated. 

Student interaction behavior was evaluated for each dyad in both types of instructional settings. In the 
face-to-face conditions, dyads were observed by the principal researcher and a research assistant using an 
observation sheet. Each instance of collaboration was recorded on the observation form. Raters identified each 
dyad by code and were unaware of the dyad’s type of cued condition to avoid any bias. The observation sheet was 
adapted from versions used in previous collaborative learning studies (Klein and Pridemore, 1992; Sherman, 1994; 
Tutty, 2006) and was used to record the different types of student behavior during the interaction screens (discussion 
checkpoints). Some of these categories of behavior included summarizing, discussing/explaining, asking for help, 
and identifying errors. These interaction behaviors related were grouped under one category as on-task behavior and 
defined as students discussing only content related to instruction. The other category, off-task behavior, comprised 
of other behaviors that were observed and recorded and included any incidents of students being off-task during 
discussion points. Off-task behavior included talking to members of other dyads and talking to their partners about 
things unrelated to the instructional content. Each dyad was observed in two minute intervals at various points 
throughout the instruction comprising a minimum of 25 minutes of observation time. 

Student interaction in the online condition was recorded using the virtual classroom session log feature of 
Blackboard. The log files for each dyad were exported to a database for analysis by the primary researcher, and 
were examined for the same type of collaborative behaviors as the dyads in the face-to-face conditions. To ensure 
an equivalent comparison of interactions between the face-to-face and online dyads, time stamps for each interaction 
in the online chatroom were used to reconstruct the class session on a timeline. Two-minute intervals were then 
systematically categorized to simulate the observations of the face-to-face dyads. The principal researcher 
systematically placed each online transcript for each dyad on a physical layout of the timeline of the class session, 
according to each time stamp. Eor example, dyads 1 and 2 were observed under the 0-2 minute timestamp, dyads 3 
and 4 were observed under the 2-4 minute timestamp, dyads 5 and 6 were observed under the 4-6 minute timestamp, 
and so forth. The principal researcher then systematically moved each pair of dyads’ online transcripts accordingly 
as they moved across the timeline. 



Results 



Posttest Performance 

The average posttest score for all participants was 16.18 (SD = 3.01) out of a possible score of 25. The 
mean posttest score was 16.51 (SD = 3.06) for students in the face-to-face collaborative condition and 15.91 (SD = 
2.98) for students in the online collaborative condition. Eurthermore, the mean posttest score was 16.61 (SD = 2.97) 
for students who received non-scripted cues and 15.76 (SD = 3.02) for students who received scripted cues. A 2 X 2 
analysis of variance (ANOVA) was conducted to determine the effect of interaction cues and instructional setting on 
individual posttest performance. ANOVA did not reveal a significant main effect for interaction cues, /'’(I, 78) = 
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1.74, p = A9, partial rf = .02 and instructional setting, F(l, 78) = .75, p = .39, partial rf = .01. In addition, there was 
a nonsignificant interaction between interaction cues and instructional setting, /'’(I, 78) = .44, p = .51, partial = 

. 01 . 

Group Project Performance 

The mean project score for all dyads was 24.83 (SD = 4.27) out of a possible score of 30. The mean project 
score was 23.86 (SD = 3.82) for dyads in the face-to-face instructional setting condition and 25.45 (SD = 4.51) for 
dyads in the online instructional setting condition. The mean group project score was 24.50 (SD = 4.06) for students 
in the dyads that received non-scripted cues and 25.17 (SD = 4.57) for students in the dyads that received scripted 
cues. A 2 X 2 analysis of variance (ANOVA) was conducted to evaluate the effect of interaction cues and 
instructional setting on group application project performance. ANOVA indicated a nonsignificant effect for 
interaction cues, F(l,32) = .11, p= .74, partial rf = .00. The ANOVA also revealed a nonsignificant effect for 
instructional setting, F(l,32) = 1.15, p = .29, partial rf = .04 There was no significant interaction between 
interaction cues and instructional setting. 

Student Attitudes 

Attitude scores were based on a 5 -point, Likert- type scale (ranging from 5 - strongly agree to 1 - strongly 
disagree). These data indicated that most students felt better prepared to use spreadsheets after completing the 
instructional program (M = 3.51, SD = 0.99) and that the program was easy to navigate (M = 3.43, SD = 1.06). 

Most students thought the instruction was easy to understand (M = 3.34, SD = 1.10) and that the computer program 
was a good way to learn about spreadsheets (M = 3.33, SD = 1.06). Many felt that they knew when to interact with 
their partner throughout the program (M = 3.59, SD = 0.97) and that they were able to use the information they 
learned from the computer program to do the practice and group projects (M = 3.39, SD = 1.09). However, students 
did not feel that they learned the material better working with a partner than they would have on their own (M = 

2.43, SD = 1.14) and did not respond favorably to preferring to working with a partner over working alone during 
the spreadsheet lesson (M = 2.70, SD = 1.28). 

MANOVA conducted using all fifteen survey items as the dependent variables indicated a significant main 
effect for instructional setting, Wilks’ A= 0.53, F(3, 66) = 3.07, p <.001. MANOVA did not show a significant 
main effect for interaction cues or an interaction between cues and instructional setting. Follow-up univariate 
analyses indicated a significant difference between instructional settings on nine of the fifteen items (p < .05). In all 
cases, students in the face-to-face setting responded more positively toward the program and collaboration method 
than those in the online instructional setting. 

Approximately 98.5% of the study participants who completed the Likert portion of the attitude survey also 
responded to the three open-ended questions. Eleven participants indicated that they liked how the objectives were 
presented, nine mentioned the interactivity in the CBl, and eight mentioned that the practice was helpful. Four 
participants explained that the CBl and collaboration strategy was a new way of learning. When asked what they 
liked least about the program, 22 respondents (32%) indicated that they had difficulty working with a partner. 
Furthermore, 12 participants mentioned they did not like using the chat room feature, and 10 felt that there was not 
enough time to complete the CBl, practice and group activity. 

Finally, when asked about how to improve the program, 12 out of 69 respondents indicated that the 
program could be improved by not using partners, 12 mentioned not using chat, and 1 1 preferred better navigation 
and/or more information presented on each screen. A participant in the online condition summed up their feelings 
toward using the chat feature for collaboration by saying, “Make it easier for communication. . .maybe have the 
partners sit next to one another so that each partner will get all the information being given.” 

Student Interviews 

A sample of 1 1 students from the face-to-face condition and 1 1 students from the online condition 
participated in an interview to determine their opinions of the program (n = 22). Nine out of the 1 1 students in the 
face-to-face instructional setting and all 1 1 students in the online setting indicated they liked the program. Other 
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students said they liked the program because of the step-by-step presentation of content, or because of the 
graphically displayed examples. When asked about which parts of the CBI were most helpful, 6 of 1 1 students in 
the face-to-face condition and 7 of 1 1 students in the online condition indicated that the individual instruction 
component was the most helpful. Two students in the face-to-face condition said that the navigation was the most 
helpful, while two in the face-to-face condition thought that the practice activity helped them the most. 

When asked about which components of the CBI were least helpful, three students in the face-to-face 
condition revealed that utilizing the different tracks was the least helpful part, while six students in the online 
condition identified the use of the Internet chat room as the least helpful. Responses to questions about working 
with a partner revealed that all 1 1 students in the face-to-face instructional setting and 8 out of 1 1 students in the 
online instructional setting liked working with a partner to learn about spreadsheets. 

When asked about how the lesson was used to learn about spreadsheets, 11 of 22 students responded that 
they followed the instructions. Three of the 1 1 respondents in the face-to-face instructional setting and 4 of the 1 1 
students in the online condition mentioned learning from their partner. Finally, respondents were asked about their 
opinion about using a similar CBI and collaborative strategy in the future. Eight of the 1 1 students in the face-to- 
face instructional setting and 9 of 1 1 students in the online instructional setting indicated that they thought it would 
be a favorable idea to use a similar lesson in the future. One student explained, “I liked the program. It’s nice to 
work on your own tutorial of skills and it was easy for me to explain to my partner what I learned.” Conversely, 
three students in the face-to-face condition and two in the online condition thought it would be a bad idea to use a 
similar program in the future. For example, one student said, “There needs to be effective communication with your 
partner. Since we’re in a teaching environment, we have the habit of doing our own work. You would need to 
introduce this learning with partners using computers at the beginning of the semester.” 

Student Interactions 

There were two categories of interaction behaviors: on-task (discussion of instructional content) and off- 
task. Separate chi-square analyses were conducted to determine the effect of instructional setting and interaction 
cues on the frequency of interactions for each category. The results of the analyses indicated that students in the 
face-to-face collaborative condition exhibited more on-task behavior while interacting with their partners during 
instruction, (1, N = 10) = 251.9, p < .05, than those in the online collaborative condition. However, students in 
the online instructional setting exhibited more off-task behavior than those in the face-to-face instructional setting, 
X\l, N = 10) = 8.91, p < .05. Chi-square analyses also revealed that students who received non-scripted cues 
exhibited more off-task behavior than those who received scripted cues, X^{1, N = 10) = 5.56, p < .05. No 
significant differences were found between the non-scripted cues and scripted cues treatments for on-task behavior. 

Time on Task 

The average total amount of time spent in the program was 204.88 minutes for dyads in the face-to-face 
instructional setting and 221.61 minutes for dyads in the online instructional setting. The total time in the program 
was 210.61 minutes for the dyads who received non-scripted cues and 217.84 minutes for dyads who received 
scripted cues. A 2 X 2 MANOVA was conducted to determine the effect of interaction cues and instructional 
setting on the time spent on orientation screens, instruction, and the practice project. This analysis indicated a 
significant main effect for instructional setting, Wilks’ A = .56, F{3, 35) = 8.81, p <.001. The results of the 
MANOVA did not show a significant main effect for interaction cues or an interaction between interaction cues and 
instructional setting. 



Conclusion 

Results for individual posttest achievement and group project performance did not reveal a significant 
effect for instructional setting or interaction cues. A plausible explanation for these findings is that participants in all 
treatment conditions used computer-based instruction designed following a systems approach. The CBI included an 
orienting activity (a list of instructional objectives for the lesson), practice (developing a gradebook), and feedback 
(model examples of what the finished gradebook should look like). Thus, all of the elements of effective instruction 
were provided to all participants in this study. Consequently, the treatment variables under investigation may not 
have been strong enough to impact performance outcomes in this study. 
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Other researchers have suggested that the use of well designed instruction can override the impact of 
treatment variables in cooperative learning studies (Klein & Pridemore, 1994). McLoughlin and Luca (2002) 
suggested that researchers should consider the different tasks inherent in face-to-face and online collaborative 
environments. For example, in face-to-face classrooms, students collaborate by engaging in tasks that are 
specifically defined and feedback on learning is readily available. In contrast, learning in online environments is 
often limited to presentation of visual information and there is often a lack of feedback (McLoughlin & Luca, 2002). 
In the current study, students in both the face-to-face and online settings received embedded interaction prompts as 
they completed a designated task. Feedback was readily available to all students. 

Results for interaction cues in the current study are inconsistent with the results of other studies that have 
effectively implemented cues during computer-based, collaborative instruction (Hooper, Temiyakarn, & Williams, 
1993; McDonald et ak, 1985; Sherman & Klein, 1995). This likely occurred because all treatment groups in this 
study were prompted to interact, whether they received scripted or non-scripted cues. This is in contrast to previous 
studies that have determined significant effects of interaction cues on performance between groups who received 
verbal interaction cues and those who did not receive any cues at all (Fletcher, 1985; Sherman & Klein,1995). 

Results for attitudes are consistent with those reported by other researchers who have studied collaboration 
in online settings (Tutty & Klein, 2008; Uribe et ak, 2003). In this study, participants were generally positive about 
the computer program and the instructional content. Most students felt that the computer program was a good way 
to learn about spreadsheets and that it was easy to use and understand. Responses to the interview questions revealed 
that many students liked the computer program because of the step-by-step presentation of content and the 
graphically displayed examples. 

However student attitudes toward collaboration were less positive. Survey data revealed that most students 
did not feel that they learned the material better working with a partner, and overall, preferred working alone during 
the spreadsheet lesson. These results were supported by student responses to the open-ended questions stating that 
they had difficulty working with a partner, especially when using the chat room feature in the online setting. 

Findings revealed that participants in the face-to-face condition had significantly more positive attitudes 
than those in the online condition. Results from the attitude survey indicated significant differences in favor of face- 
to-face students on nine of the fifteen survey items. Thus, students in the face-to-face setting thought they were able 
to adequately communicate with their partner, and knew how and when to interact with them during instruction 
more than those who were in the online condition. 

Analyses of student interaction data revealed that students in the face-to-face setting exhibited significantly 
more on-task behavior during instruction than those in the online setting. This finding is certainly not surprising, 
considering the tendency for students in a face-to-face setting to interact more naturally and easily. In contrast, 
students in the online condition displayed significantly more off-task behavior. These results likely occurred 
because it may have been due to the novelty of the learning environment, as well as the use of the chat room feature. 
Typically, interaction in a chat room more often occurs in a casual setting, and mainly involves discussion of topics 
unrelated to academic or learning purposes. This is supported by the nuances revealed by a content analysis of chat 
room transcripts, where there were many instances of students discussing topics unrelated to the instructional 
content. Students expressed frustration toward using the chat room to work together on an instructional task. This is 
further supported by interview data that revealed that online collaboration was difficult and that it was challenging to 
teach their partner computer skills through a virtual environment because they were unable to physically show them 
how to do it. 

It is likely that the increased off-task behaviors for students in the online setting are a result of their need to 
establish a social presence. Online students often feel that it is necessary to establish a degree of comfort and safety 
before expressing their ideas in a collaborative context (Anderson, 2004; Laister & Kober, 2002). The results of 
this study were similar to a recent study by Tutty & Klein (2008) where virtual dyads exhibited significantly more 
off-task interactions than face-to-face dyads. 

Time data revealed that students in the online instructional setting spent significantly more time on 
orientation screens than the students in the face-to-face condition. Analysis of the chat transcripts of dyads in the 
online condition indicated many instances where students experienced extended wait time for their partners during 




the beginning of instruction because of unforeseen technical issues. In these cases, the Blackboard course 
management tool kept inadvertently logging off students from the system, causing them to log back on again. 

Another plausible explanation for the significantly longer time spent on orientation screens is that students 
in the online condition had to simultaneously work with three open windows — the CBI program. Excel, and the chat 
feature. Thus, users had to constantly switch back and forth between windows during instruction and collaboration. 
This multitasking might have impacted the time it took for individual students to orient themselves to this non- 
traditional learning situation, as well as their retention and learning of the content itself. When this occurred, it may 
be classified as a split-attention effect when the learner’s attention must be split between multiple sources of visual 
information that are essential for understanding (Sweller, 2005). Having to mentally process multiple sources of 
information may have imposed an extraneous cognitive load on the participants, and hence, may explain why there 
lacked an optimal understanding and learning of the instructional material. 

Implications 

Participants in this study found it difficult and frustrating to complete the group application project because 
they felt they did not have enough time to do so. Interview data revealed that students wanted more time to 
effectively communicate with their partners. This suggests that collaborative learning strategies should be 
introduced and practiced for a greater length of time before implementing a collaborative computer lesson. 
Furthermore, the relatively short duration of the treatment may have influenced outcomes in the current study. 
Extending the time for overall instruction could produce different results for performance outcomes. 

The results of this study also imply that the systematic application of appropriate instructional elements 
increases the likelihood of successful learning in computer-based collaborative settings. While employing well 
designed instruction may not produce significant differences between treatment variables in collaborative learning 
research, the elements of effective instruction will have a positive impact on outcomes such as posttest achievement 
and project performance. 

Future Research 

Future research should investigate the effects of using a chat room built into the same CBI to determine if 
students would learn the content better and communicate more effectively with less screens to contend with. 
Effective training should orient students to the use of the chat room for collaboration while using another CBI so 
that there would be less multitasking and less potential for affecting cognitive load, especially when novel 
instruction is presented to learners. For this particular study, participants in the online condition felt that the chat 
room component of the lesson was another facet that added to their unfavorable responses toward collaboration 
during instruction. 

Future research should continue to explore the effects of using a synchronous chat feature that incorporates 
more capabilities simulating a face-to-face environment. This would include examples, such as video conferencing, 
or the ability to synchronously work together on a particular task (i.e., both students in the dyad would be able to 
remotely work on a single spreadsheet within the CBI environment). This may help to further increase social 
presence in a virtual setting. 

Another suggestion for future research is to address the effects of scripted cues and no cues for face-to-face 
and online groups. The effects of structuring verbal interaction may become clearer if the group receiving scripted 
cues were compared to a group that does not receive any cues throughout an instructional program. This research 
design may help determine the best possible way to develop CBI programs with embedded interaction prompts. 

Thoughtful consideration should be taken when generalizing the results of this study until further research 
can be conducted. Specifically, more research on the design of interaction cues that would be effective during 
collaborative computer based instruction is required. Researchers must have appropriate instructional strategies in 
place to effectively support collaboration and interaction in specific instructional settings to ensure the goal of 
achieving student learning. 

For more information, email Christy M. Alarcon at Christy.Alarcon@asu.edu. 



9 




References 



Anderson, T. (2004). Teaching in an online learning context. In T. Anderson and Faith Elloum (Eds.). Theory and 
practice of online learning (pp. 273-294). Athabasca, Canada: Athabasca University. 

Blocher, M. (2005). Increasing learner interaction: Using Jigsaw online. Educational Media International, 42(3), 
269-278. 

Brewer, S. A. (2004). Small group learning in an online asynchronous environment. Unpublished Dissertation, 
Arizona State University, Tempe, (UMI No. 3123524). 

Brewer, S. A., & Klein, J. D. (2006). Type of positive interdependence and affiliation motive in an asynchronous, 
collaborative learning environment. Educational Technology Research and Development, 54(4), 331-354. 

Brewer, S. A., Klein, J. D., & Mann, K. E. (2003). Using small group learning strategies with adult re-entry 
students. College Student Journal, 37(2), 286-297. 

Eletcher, B. (1985). Group and individual learning of junior high school children on a micro-computer-based task. 
Educational Review, 37, 252-261. 

Hooper, S. (1992). Cooperative learning and computer-based instruction. Educational Technology Research and 
Development, 40(3), 21-38. 

Hooper, S. (1992). Effects of peer interaction during computer-based mathematics instruction. Journal of 
Educational Research, S5(3), 180-189. 

Hooper, S., & Hannafin, M. J. (1991). The effects of group composition on achievement, interaction, and learning 
efficiency during computer-based cooperative instruction. Educational Technology Research and 
Development, 39(3), 27-40. 

Hooper, S., Temiyakarn, C., & Williams, M. D. (1993). The effects of cooperative learning and learner control on 
high- and average-ability students. Educational Technology Research and Development, 41(2), 5-18. 

Hudson, B. (1996). Group work with multimedia in mathematics: Contrasting patterns of interaction. British Journal 
of Educational Technology, 27, 171-190 

Johnson, D. W., & Johnson, R. T. (2004). Handbook of research on educational communications and technology. 
Retrieved November 13, 2006, from http://www.aect.org/edtech/default.asp . 

Klein, J. D., Erchul, J. A., & Pridemore, D. R. (1994). Effects of individual versus cooperative learning and type of 
reward on performance and continuing motivation. Contemporary Educational Psychology, 19, 24-32. 

Klein, J. D., & Pridemore, D. R. (1992). Effects of cooperative learning and need for affiliation on performance, 
time on task and satisfaction. Educational Technology Research and Development, 40(4), 39-41 . 

Klein, J. D., & Pridemore, D. R. (1994). Effects of orienting activities and practice on achievement, continuing 
motivation, and student behaviors in a cooperative learning environment. Educational Technology 
Research and Development, 42(4), 41-54. 

Ku, H.-Y., Lohr, L., & Cheng, Y.-C. (2004). Collaborative learning experiences in online instructional design 

courses. Paper presented at the 27* Annual Meeting of the Association for Educational Communications 
and Technology, Chicago, IE. 



10 



Laister, J., Kober, S., & Joanneum, T. (2002). Social aspects of collaborative learning in virtual learning 

environments. Paper presented at the 2002 Networked Learning Conference. Abstract retrieved April 12, 
2008, from http://comma.doc.ic.ac.uk/inverse/papers/patras/19.pdf 

Mackie, B., & Gutierrez, C. F. (2005). Using pictures as a vehicle to personalize the collaborative learning 
environment. J. Educational Technology Systems, 33(3), 189-204. 

McDonald, B. A., Larson, C. O., Dansereau, D. F., & Spurlin, J. E. (1985). Cooperative dyads: Impact on text 
learning and transfer. Contemporary Educational Psychology, 10, 361-311 . 

McLoughlin, C., & Luca, J. (2002). A learner-centred approach to developing team skills through web-based 
learning and assessment. British Journal of Educational Technology, 5(5), 571-582. 

Pollard, C., VanDehey, T., & Pollard, R. (2003). Curricular computing: Essential skills for teachers. Boise, ID. 
Boise State University Printing. 

Reeves, T. C., Herrington, J., & Oliver, R. (2004). A development research agenda for online collaborative learning. 
Educational Technology, Research and Development, 52(4), 53-65. 

Sherman, G. P. (1994). The effects of cued interaction and ability grouping during cooperative computer-based 
science instruction. Unpublished Dissertation, Arizona State University, Tempe, (UMI No. 9500740). 

Sherman, G. P., & Klein, J. D. (1995). The effects of cued interaction and ability grouping during cooperative 
computer-based science instruction. Proceedings of the Annual Meeting of the American Educational 
Research Association, San Francisco, CA, 2-36. 

Sherman, G. P., & Klein, J. D. (1995). The effects of cued interaction and ability grouping during cooperative 
computer-based science instruction. Educational Technology Research and Development, 43(4), 5-24. 

Skiba, D. J. (2006). Collaborative tools for the net generation. Emerging Technologies Center, 27(3). 

Sweller, J. (2005). Implications of cognitive load theory for multimedia learning. In R.E. Mayer (Ed.), The 

Cambridge handbook of multimedia learning (pp. 19-30). New York, NY: Cambridge University Press. 

Tutty, J. I. (2006). Effects of collaboration mode and group composition in computer-mediated instruction. 
Unpublished Dissertation, Arizona State University, Tempe. 

Tutty, J. I., & Klein, J. D. (2008). Computer-mediated instruction: A comparison of online and face-to-face 
collaboration. Educational Technology Research & Development, 56(2), 101-124. 

Uribe, D., Klein, J. D., & Sullivan, H. (2003). The effect of computer-mediated collaborative learning on solving ill- 
defined problems. Educational Technology Research & Development, 57(1), 5-19. 



11 



Beliefs, Emotions, and Academic Success: Viewing Online Distance Learning 
Through the Lens of Social Cognitive Theory 
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Abstract 

This study examined online distance learning from a social cognitive perspective. Participants (N = 481) 
completed a survey that assessed their motivational beliefs (self-efficacy and task value); negative achievement 
emotions (boredom and frustration); and several outcomes that included their use of self-regulated learning 
strategies (elaboration and metacognition), course satisfaction, and continuing motivation to enroll in future 
online courses. Results from several multiple regressions revealed that task value beliefs were the strongest and 
most consistent positive predictors of elaboration, metacognition, satisfaction, and continuing motivation; 
whereas self-efficacy beliefs were moderately strong positive predictors of satisfaction and continuing 
motivation only. On the other hand, students’ boredom and frustration were statistically significant predictors of 
metacognition, with boredom emerging as a negative predictor and frustration unexpectedly emerging as a 
positive predictor. Furthermore, both boredom and frustration were negatively related to satisfaction and 
continuing motivation. Taken together, results from this study provide some insight into the complex relations 
between personal and behavioral influences on self-regulated learning and overall academic success in an online 
course. Implications for the theory, research, and practice of academic self-regulation and online learning are 
discussed, as are study limitations and future directions. 

Background 

Educational psychologists have long known that students who are motivated to learn experience greater 
academic success than those who are not (Schunk, Pintrich, & Meece, 2008). One reason motivated students 
succeed is they engage in self-regulated learning behaviors that help make learning more efficient (Flavell, 1979; 
Schunk et ah, 2008). Simply stated, most academically successful students are highly-motivated, self-regulated 
learners (Pintrich, 2003). 

Unfortunately, not all students are highly-motivated, self-regulated learners. Many students fail to see the 
value of what they are learning or do not feel competent enough to master what is being taught (Pintrich, 1999); 
some are too bored, frustrated, and/or anxious to ever become academically engaged (Pekrun, Goetz, Titz, & Perry, 
2002); and still others fail to use effective learning strategies (Pintrich & De Groot, 1990). And while students like 
these may struggle in traditional classrooms, when faced with learning online, they may be even more 
disadvantaged. This increased difficulty occurs because learning on the web is thought to be a highly autonomous 
endeavor that requires considerable self-direction (Dabbagh & Kitsantas, 2004). 

Purpose of the Study 

Utilizing social cognitive theory as a framework (Zimmerman, 2000), this study examined the importance 
of students’ motivational beliefs and negative achievement emotions in explaining their self-regulated learning 
behaviors and academic success. In doing so, this study answers recent calls (e.g., Bernard et ah, 2004) to move 
beyond group comparison studies in online learning research and, instead, to focus on the attributes, skills, 
behaviors, and attitudes that contribute to success in online settings. 

In particular, two research questions were addressed in this study: RQl - After controlling for demographic 
and experiential variables, how are students’ personal motivational beliefs (self-efficacy and task value) and 
negative achievement emotions (boredom and frustration) related to their self-reported use of elaboration and 
metacognition? RQ2 - After controlling for demographic and experiential variables, how are students’ personal 
motivational beliefs and negative achievement emotions related to their academic success, as measured by their 
satisfaction with the online course and continuing motivation to take future online courses? 
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Method 



Participants, Materials, and Procedures 

A convenience sample of 481 undergraduates (sophomores and juniors) from a U.S. service academy were 
invited to participate in this study. The sample included 398 men (83%) and 83 women (17%). The mean age of the 
participants was 20.5 years {SD = 1.0; range 19-24). 

The instructional materials consisted of a self-paced online course developed by the U.S. Navy. Self-paced 
online courses are a specific type of online training in which students use a web browser to access a course 
management system and complete web-based instruction at their own pace. While completing these courses, 
students do not interact with an instructor or other students. 

The online course was the first part of a two-stage training program in flight physiology and aviation 
survival training that was required for all service academy undergraduates. Upon successful completion of the online 
course, students advanced to the second stage of their training, which consisted of traditional instruction at a local 
training unit. 

The online course was composed of four, 40-minute lessons. Each lesson included text, graphics, video, 
interactive activities, and end-of-lesson quizzes that consisted of 12 to 15 multiple-choice questions. Students who 
did not score at least 80% on any given quiz were required to return to the beginning of the lesson, review the 
material, and then retake the quiz. 

Approximately three weeks after completing the self-paced online course, students arrived at a local 
training unit for the face-to-face portion of their instruction. Prior to any classroom training, all students were invited 
to complete an anonymous, self-report survey. Participation in the survey was completely voluntary; 100 percent of 
the students completed the survey. 

Instrumentation 

The instrument used in this study was composed of 50 items divided into two sections. The first section 
included 41 Likert-type items with a response scale ranging from 1 {completely disagree) to 7 {completely agree). 
These 41 items were further subdivided into eight subscales designed to assessed students’ motivational beliefs 
(self-efficacy and task value), negative achievement emotions (boredom and frustration), use of cognitive and 
metacognitive learning strategies (elaboration and metacognition), overall course satisfaction, and self-reported prior 
knowledge. 

Motivational beliefs. Two subscales from Artino and McCoach (2008) were used to assess students’ 
personal motivational beliefs: (a) a five-item self-efficacy subscale designed to assess students’ confidence in their 
ability to learn the material presented in a self-paced online format and (b) a six-item task value subscale designed to 
assess students’ judgments of how interesting, important, and useful the online course was to them. Sample items 
from this section include “I am confident I can learn without the presence of an instructor to assist me” (self- 
efficacy) and “It was personally important for me to perform well in this course” (task value). 

Negative achievement emotions. Two subscales adapted from the Achievement Emotions Questionnaire 
(AEQ; Pekrun, Goetz, & Perry, 2005) were used to assess students’ negative achievement emotions: (a) a five-item 
boredom subscale intended to assess students’ course -related boredom and (b) a fowt-iitm frustration subscale 
designed to assess students’ course-related frustration, annoyance, and irritation. Sample items from this section 
include “While completing this online course I was bored” (boredom) and “While completing this online course I 
felt frustrated” (frustration). 

Self-regulated learning strategies. Students’ use of cognitive and metacognitive learning strategies was 
assessed with items derived from the Motivated Strategies for Learning Questionnaire (MSLQ; Pintrich, Smith, 
Garcia, & McKeachie, 1993): (a) a four-item elaboration subscale designed to assess students’ use of elaboration 
strategies (e.g., paraphrasing and summarizing) and (b) a nine-item metacognition subscale intended to assess 
students’ use of metacognitive control strategies (e.g., planning, setting goals, monitoring one’s comprehension, and 
regulating performance). The items included in this section were similar to the original MSLQ, except that some 
items were reworded to reflect the online nature of the course. Sample items include “While working through this 
online course I tried to relate what I was learning to what I already know” (elaboration) and “While working through 
this online course I set goals for myself in order to direct my activities” (metacognition). 

Satisfaction. Students’ overall satisfaction with the online course was assessed with a three-item 
satisfaction subscale adapted from Artino (2008). Sample items include “Overall, I was satisfied with my online 
learning experience” and “This online course met my needs as a learner.” 
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Prior Knowledge. Students’ self-reported prior knowledge of the online course material was measured with 
a five-item prior knowledge subscale. This subscale assess students’ familiarity with the four terminal learning 
objectives; that is, students’ appraisal of how much they knew prior to completing the online course. Sample items 
include “I could identify the methods used to prevent motion sickness in flight” and “I was familiar with how the 
different sensory systems function in flight.” 

Section two of the survey was composed of nine items, including background and demographic questions 
and three individual items used as variables in this study: 

Online technologies experience. Online technologies experience was assessed with a single self-report 
item: “Compared to other Midshipmen, how experienced are you with online computer technologies (for example, 
using a web browser, surfing the Internet, etc.)?” The response scale ranged from 1 {extremely inexperienced) to 7 
{extremely experienced). 

Online learning experience. Online learning experience was assessed with a single self-report item: 
“Compared to other Midshipmen, how experienced are you with self-paced online learning (for example, courses 
like the online portion of this course)?” Again, the response scale ranged from 1 {extremely inexperienced) to 7 
{extremely experienced). 

Continuing motivation. Continuing motivation (Maehr, 1976) to take future online courses was assessed 
with a single self-report item: “Considering your experience with this online course, would you choose to enroll in 
another self-paced online Navy course in the future? Please answer this question as if the choice were completely up 
to you.” The response scale ranged from 1 {definitely will not enroll) to 6 {definitely will enroll). 

Results 



Confirmatory Factor Analysis 

Confirmatory factor analysis (CFA) substantiated the hypothesized eight-factor structure. All model fit 
statistics fell within recommended standards (Hu & Bender, 1999): the chi square was statistically significant, 

(436, A = 471) = 860.333, p < .001; however, the normed chi square (NC, 1.97) was less than 2.00, the comparative 
fit index (CFI, .955) was slightly greater than .95, and the root-mean-square error of approximation (RMSEA, .046) 
was less than .06. 

Based on the results of the CFA, reliability analyses were run on the items retained in the eight subscales. 
As indicated in Table 1, Cronbach’s alphas for the subscale scores of the seven primary variables of interest were 
quite good (i.e., > .80; see guidelines in Gable & Wolfe, 1993). 

Descriptive Statistics and Correlation Analysis 

Descriptive statistics for the measured variables are provided in Table 1. As indicated, six of the seven 
variables measured on a 7-point Likert-type scale had means at or above the midpoint of the response scale; while 
one variable (frustration) had a mean slightly below the midpoint. The mean score for continuing motivation (3.93; 
measured on a 6-point Likert-type scale) was also above the midpoint of the response scale. Standard deviations for 
these eight variables ranged from 1.07 to 1.45, and visual inspection of the associated histograms showed that all 
variables, with the exception of frustration, were negatively skewed. The distribution for frustration showed a slight 
positive skew. 

Table 1 



Pearson Correlations, ( Cronbach ’s Alphas ), Means, and Standard Deviations for the Primary Variables of Interest 
(F\=481) 



Variable 


1 


2 


3 


4 


5 


6 


7 


8 


1. Self- 
Lfficacy 


(.91) 


.32 


-.27 


-.27 


.27 


.18 


.41 


.36 


2. Task Value 




(.88) 


-.41 


-.39 


.56 


.61 


.66 


.41 


3. Boredom 






(.84) 


.58 


-.28 


-.35 


-.52 


-.46 


4. Lrustration 








(.89) 


-.23 


-.22 


-.52 


-.43 


5. Elaboration 










(.82) 


.59 


.50 


.29 


6. 












(.89) 


.54 


.33 
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Metacognition 
7. Satisfaction 














(.92) 


.59 


8. Continuing 
Motivation 
















(-) 


M 


5.32 


4.87 


4.02 


3.36 


4.81 


4.12 


4.77 


3.93 


SD 


1.12 


1.09 


1.32 


1.45 


1.08 


1.07 


1.20 


1.17 



Note. Continuing motivation was measured on a 6-point, Likert-type response scale from 1 {definitely will not 
enroll) to 6 {definitely will enroll). All other variables were measured on a 7-point, Likert-type agreement response 
scale. All correlations are significant at the p < .001 level. 



Pearson correlations indicated that self-efficacy and task value were statistically significantly related to 
each other {r = .32, p < .001) and to students’ negative achievement emotions and several measures of academic 
success. In particular, students’ perceived self-efficacy was negatively related to their boredom {r = -.27, p < .001) 
and frustration (r = -.27, p < .001) with the online course, and positively related to their self-reported use of 
elaboration {r = .27, p < .001) and metacognition {r= .18, p < .001) strategies. Self-efficacy beliefs were also 
positively related to satisfaction (r = .41, p < .001) and continuing motivation (r = .36, p < .001). Likewise, the 
extent to which students valued the online learning tasks was negatively related to their boredom {r = -.41, p < .001) 
and frustration {r = -.39, p < .001), and positively related to their elaboration {r = .56, p < .001) and metacognition {r 
= .61, p < .001). Task value was also positively related to satisfaction {r = .66, p < .001) and continuing motivation 
(r = .41, p < .001). Overall, these results indicate that when considered individually, students’ motivational beliefs 
explained from 3% to 44% of the variance in the various measures of online academic success (weak to very strong 
effects; Cohen, 1988). 

In terms of students’ negative achievement emotions, Pearson correlations indicated that boredom and 
frustration were statistically significantly related to each other {r = .58, p < .001) and to all measures of academic 
success. In particular, students’ boredom was negatively related to their self-reported elaboration (r = -.28, p < .001), 
metacognition {r = -.35, p < .001), satisfaction {r = -.52, p < .001), and continuing motivation (r = -.46, p < .001). 
Likewise, the extent to which students reported being frustrated with the online course was negatively related to 
their elaboration {r = -.23 p < .001), metacognition {r = -.22, p < .001), satisfaction {r = -.52, p < .001), and 
continuing motivation (r = -.43, p < .001). Overall, these results indicate that when considered individually, 
students’ negative achievement emotions explained from 5% to 27% of the variance in their self-reported use of 
learning strategies, satisfaction, and continuing motivation (weak to strong effects). 

Multiple Regression Analysis 

To explore the unique variance explained by students’ beliefs and emotions on several adaptive outcomes, 
four multiple regressions were conducted. In these analyses, elaboration, metacognition, satisfaction, and continuing 
motivation were used as the dependent variables; self-efficacy, task value, boredom, and frustration — along with 
five control variables (gender, age, online technologies experience, online learning experience, and prior 
knowledge) — served as the independent variables. 

Table 2 

Regression Summary Statistics for the Four Dependent Variables: Elaboration, Metacognition, Satisfaction, and 



Continuing Motivation 


Elaboration 

Variable 


Metacognition 


Satisfaction 


Continuing 

Motivation 


B SEB P 


B SEB P 


B SEB P 


B SEB P 



Self- 

Efficacy 

Task Value 


.07 


.04 


.07 


-.02 


.04 


-.02 


.21 


.04 


.20* 

* 


.18 


.04 


.17* 

* 


.50 


.04 


.51* 

* 


.58 


.04 


.57** 


.51 


.04 


.46* 

!l! 


.17 


.05 


.17* 

* 


Boredom 


-.07 


.04 


-.08 


-.15 


.04 


- 18** 


-.16 


.04 


.18* 


-.20 


.04 


.22* 



* * 
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Frustration .03 .04 .04 .08 .04 .10* -.16 .03 .19* -.14 .04 .18* 

* * 



Model 

Summary 



R^=35,p<.001 



R^= 39, p< .001 



R^= .57,p< .001 R^ = .34, p < .001 



Note. Summary statistics are calculated after controlling for gender, age, online technologies experience, online 
learning experience, and prior knowledge. Continuing motivation was measured on a 6-point, Likert-type response 
scale from 1 (definitely will not enroll) to 6 (definitely will enroll). All other variables were measured on a 7-point, 
Likert-type agreement response scale. 

*p < .05; **p < .001. 



Results revealed that all four models explained statistically significant amounts of variance in the 
outcomes. Model effects were strong, ranging from R^ = .34 for continuing motivation to R^ = .57 for satisfaction. 
Examination of the standardized beta coefficients suggests that task value was the strongest and most consistent 
individual predictor. Specifically, task value was a positive predictor of elaboration (P = .51, p < .001), 
metacognition (P = .51, p < .001), satisfaction (P = .46, p < .001), and continuing motivation (P = .17,p < .001). 
Self-efficacy was a significant positive predictor of satisfaction (P = .20, p < .001) and continuing motivation (P = 
.17,p < .001) only. In terms of negative achievement emotions, both boredom and frustration were significant 
negative predictors of satisfaction (P = -.18 and -.19, respectively) and continuing motivation (P = -.22 and -.18, 
respectively); whereas boredom (P = -18, p < .001) emerged as a negative predictor of metacognition, and frustration 
(P = .10, p < .05) unexpectedly emerged as a positive predictor of metacognition. 



Discussion 



The present study examined students’ thoughts, feelings, and actions in the context of a self-paced online 
course. In particular, this study employed a social cognitive view of self-regulation to investigate how motivational 
beliefs and negative achievement emotions relate to several adaptive outcomes. These outcomes included students’ 
use of cognitive and metacognitive learning strategies, overall satisfaction with an online course, and continuing 
motivation to take future online courses. 

Taken together, findings from this study are largely congruent with prior theory and research in the fields 
of academic motivation, self-regulation, and online learning (e.g., Artino & Stephens, 2006; Pekrun et ah, 2002; 
Pintrich, 1999; Zusho et ah, 2003). Specifically, these results indicate that the motivational beliefs and negative 
achievement emotions of students learning in a self-paced online environment are related, in important ways, to 
several adaptive academic outcomes. 

Research Question 1 

Research question 1 addressed how students’ motivational beliefs and negative achievement emotions 
relate to their use of self-regulated learning strategies in an online course. 

Motivational beliefs. Social cognitive theories of self-regulated learning stress the importance of students’ 
motivational beliefs in all phases of self-regulation (Schunk & Zimmerman, 2008). From this theoretical 
perspective, it is not enough for students to have knowledge of cognitive and metacognitive strategies; they must 
also be motivated to effectively utilize those strategies to improve learning and performance (Pintrich & De Groot, 
1990). Findings from this study generally support this view. Specifically, task value beliefs were the strongest and 
most consistent predictors of both elaboration and metacognition (Ps = .51 and .57, respectively). It appears that 
students who believed the course was interesting, important, and useful also reported using more learning strategies. 
This finding is consistent with prior research in both traditional (Pintrich, 1999; Pintrich & De Groot, 1990; Pintrich 
et ah, 1993; Zusho et ah, 2003) and online settings (Artino & Stephens, 2006; Hsu, 1997). Moreover, one might 
speculate that the consistency and strength of the relationship between task value and both elaboration and 
metacognition suggest that positive task value beliefs may be critical in online learning situations. That is, in highly 
autonomous contexts where students do not interact with an instructor or other students, adaptive motivational 
beliefs, such as the extent to which students value a course, may be vital for initiating and sustaining cognitive and 
metacognitive engagement (Zimmerman & Tsikalas, 2005). Certainly, however, more controlled studies that utilize 
longitudinal or experimental designs are needed to ultimately determine the direction of influence between task 
value beliefs and learning strategies use. 
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The associations between students’ self-efficacy and their learning strategies use were weaker and less 
consistent than those described above. In particular, when considered alone, students’ self-efficacy for learning in a 
self-paced online format was positively correlated with both elaboration and metacognition (rs = .27 and .18, 
respectively), although the effects were weak. However, after accounting for the other predictors in the two 
regression models, self-efficacy beliefs did not add unique information to the prediction of either elaboration or 
metacognition. One explanation for this unexpected finding is the somewhat general nature of the self-efficacy scale 
used in this study. As Pajares (1996) cautioned, “because judgments of self-efficacy are task and domain specific, 
global or inappropriately defined self-efficacy assessments weaken effects’’ (p. 547). Therefore, a researcher 
attempting to explain an academic outcome, for instance, is more likely to find a strong relationship between self- 
efficacy and the outcome if the efficacy scale follows two theoretical guidelines: (a) it assesses specific aspects of 
the task and (b) the specificity corresponds to the characteristics of the task being assessed and the domain of 
functioning being analyzed (Bandura, 1997). Accordingly, omnibus measures of general, contextless dispositions 
have relatively weak predictive power; whereas domain-linked measures of self-efficacy tend to be good predictors 
of numerous academic outcomes (Bandura, 1997; Pajares, 1996). 

Although the self-efficacy scale employed in this study is certainly not contextless, it is broad, particularly 
when compared to other self-efficacy scales that have been used to measure students’ confidence for completing 
specific tasks in very narrow academic domains (e.g., a scale for measuring adolescents’ algebra self-efficacy; 
Bandura, 2006). Therefore, it is not completely surprising to find that this rather broad self-efficacy scale did not 
explain unique variance in either elaboration or metacognition. However, while the measure did not explain unique 
variance in students’ self-reported elaboration and metacognition, it does appear to have explanatory power with 
respect to students’ satisfaction and continuing motivation to enroll in future online courses (see the discussion of 
research question 2 below). Nonetheless, future research should consider the extent to which other, more domain- 
specific self-efficacy scales might better explain students’ self-regulatory behaviors in online settings. 

Achievement emotions. Along with motivational beliefs, social cognitive theories have recently addressed 
the importance of achievement-related emotions and their influence on cognitive engagement and learning 
(Linnenbrink & Pintrich, 2002, 2004; Pekrun, 2006; Pekrun et ak, 2002). In general, social cognitive, control-value 
theory assumes that “activating positive emotions facilitate the use of flexible, creative learning strategies, and 
activating negative emotions (e.g., anxiety) more rigid strategies like simple rehearsal. Deactivating emotions (e.g., 
boredom) are held to lead to superficial, shallow ways of processing information’’ (Pekrun, 2006, p. 326). Overall, 
findings from this study largely support these theoretical assumptions. Specifically, when considered individually, 
both boredom and frustration were negatively correlated with elaboration and metacognition (rs ranged from -.22 to 
-.35), indicating that students who were bored and/or frustrated where less likely to employ these adaptive learning 
strategies. However, after accounting for the other variables in the regression model, neither boredom nor frustration 
was a statistically significant predictor of elaboration. In contrast, both boredom and frustration were statistically 
significant predictors of metacognition, with boredom emerging as a negative predictor (P = -.18) and frustration 
unexpectedly emerging as a positive predictor (P = .10). In all cases, correlation and regression coefficients were 
larger for boredom, a negative deactivating emotion, than for frustration, a negative activating emotion, a finding 
that aligns with previous research using negative achievement emotions to predict learning strategies use in 
traditional classrooms (Pekrun et ak, 2002). Moreover, the finding that boredom effects were consistently stronger 
than frustration effects is consistent with recent empirical work which suggests that boredom may be one of the most 
frequently experienced and deleterious emotions in academic settings, yet one that has received very little attention 
by motivation and emotion researchers (Pekrun, Hall, & Perry, 2008). 

Although it is possible that the positive relation between frustration and metacognition found in this study 
is a statistical anomaly, the result does seem to warrant further consideration. For example, Wosnitza and Volet 
(2005) theorized that “in a solo online-learning environment, emotions are typically directed at the self, the task, or 
the technology” (p. 455). In this study, although it is unclear exactly why students reported being frustrated, one 
might speculate that feelings of frustration were likely self directed (e.g., frustration because the learner had 
difficulty understanding the material), task directed (e.g., frustration because the task was unclear), and/or 
technology directed (e.g., frustration because of problems with the course management system and/or Internet 
connectivity; Wosnitza & Volet, 2005). If, for example, students experienced self-directed frustration because they 
struggled with the course material and the way in which it was presented, it would make sense that these individuals 
might also report using more metacognitive control strategies in an effort to improve comprehension. Close 
inspection of two survey items from the metacognition subscale clarify this point: (a) If I became confused about 
something I read, I went back and tried to figure it out, and (b) If course material was difficult to understand, I 
changed the way I studied it. 
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The scenario described above is just one possible explanation for why frustration positively predicted 
metacognition. Although inconsistent with the empirical work of Pekrun and his colleagues (e.g., Goetz, Pekrun, 
Hall, & Haag, 2006; Pekrun et ak, 2002), this novel result corroborates the theoretical suggestion that “negative 
activating emotions may well facilitate the use of specific kinds of learning strategies, even if such effects do not 
appear in more consistent ways when self-report measures of learning strategies are used” (Pekrun et ak, 2002, p. 
99). Moreover, this unexpected result suggests that some amount of frustration during online learning may actually 
promote metacognitive engagement. This, however, is a very tenuous proposal and is certainly not meant to imply 
that courses should be designed to intentionally frustrate their learners. If nothing else, this novel finding is further 
evidence of the multifaceted, dynamic interplay between cognition, affect, and behavior (Linnenbrink & Pintrich, 
2004). 

Research Question 2 

Research question 2 addressed how students’ motivational beliefs and negative achievement emotions 
relate to their overall satisfaction with the online course and continuing motivation to take future online courses. 

Motivational beliefs. In addition to predicting cognitive and metacognitive engagement, task value and self- 
efficacy beliefs are thought to influence other important academic outcomes (Bandura, 1997; Eccles & Wigfield, 
2002; Pajares, 1996; Schunk & Zimmerman, 2008). For instance, previous research in online settings has shown that 
students who find a learning activity interesting, important, and useful, as well as those who are confident they can 
perform the actions necessary to attain their goals, tend to be more satisfied and motivated than their counterparts 
with less-adaptive beliefs (e.g., Artino, 2007, 2008; Lee, 2002). Moreover, several other studies (e.g., Joo, Bong, & 
Choi, 2000; Lynch & Dembo, 2004; Wang & Newlin, 2002) have revealed that positive task value and self-efficacy 
beliefs are associated with superior academic achievement. Findings from this study corroborate this empirical 
work. Specifically, results from two multiple regressions indicate that task value was the strongest positive predictor 
of satisfaction (P = .46) and a moderately strong positive predictor of continuing motivation (P = .16); whereas self- 
efficacy for learning online emerged as a moderately strong positive predictor of both satisfaction and continuing 
motivation (Ps = .19 and .17, respectively). 

Achievement emotions. Control-value theory (Pekrun, 2006) proposes that achievement emotions impact 
learning and performance through their influence on such factors as attention, motivation and effort, and use of 
cognitive and metacognitive learning strategies. The present findings support and extend this theory by providing 
evidence that achievement emotions are also related to other important outcomes, including satisfaction with an 
online course and continuing motivation to take future online courses. In particular, both negative achievement 
emotions were consistently negatively related to satisfaction and continuing motivation (rs ranged from -.43 to -.52). 
Furthermore, after accounting for the other variables in the regression model, boredom and frustration were the 
strongest individual predictors of continuing motivation (P = -.22 and -.18, respectively). These findings indicate 
that students’ satisfaction and intentions to enroll in future online courses may be closely linked to their negative 
achievement emotions. These results are important if one considers that, from a self-regulated learning perspective, 
self-reflective reactions, such as satisfaction, are thought to “influence one’s forethought regarding subsequent 
efforts to learn in cyclical fashion” (Zimmerman & Tsikalas, 2005, p. 267). Additionally, these results provide 
further evidence of the close connection between affect and motivation (Linnenbrink & Pintrich, 2004; Pekrun, 
2006). Specifically, because learner motivation cannot be observed directly, continuing motivation is often used to 
infer the presence of motivation (Eccles & Wigfield, 2002; Maehr, 1976; Schunk et ak, 2008). Thus, it appears that 
students who were bored and/or frustrated with the online course were also less motivated. 

Educational Implications 

Due to the correlational nature of this study, strong implications for online learning are somewhat difficult 
to draw. Nevertheless, results from this study provide course developers and policy makers with some insight into 
the thoughts, feelings, and actions of successful online learners. From a practical standpoint, institutions currently 
using, or planning to use, online learning may be able to utilize this information to improve their students’ overall 
experience and academic performance in self-paced online learning situations. 

Although tentative, results from this study suggest the following preliminary implications for online course 
developers and policy makers: 

Use self-regulated learning theory as a framework. Online learning has finally reached a more mature stage 
and is now considered a viable alternative to traditional classroom instruction (Tallent-Runnels et ak, 2006). 
Accordingly, comparative research is being replaced by investigations, such as this study, which attempt to explain 
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learning effectiveness and expand learning theory into online contexts. In particular, findings from this study reveal 
the importance of students’ motivational beliefs and negative achievement emotions in explaining their self- 
regulation and academic success in self-paced online situations. Thus, it seems that social cognitive models of self- 
regulation may be useful to both researchers and practitioners as they strive to better understand how students go 
about learning in online environments. 

Promote task value beliefs. Instructional designers should consider designing their online courses in such a 
way that enhances the extent to which students value the learning tasks, which can ultimately improve motivation 
and learning (Keller, 1999). For example, by clarifying the relevance of specific learning activities, instructional 
designers help students understand the contribution of coursework to the realization of their personal goals, interests, 
and values (Assor, Kaplan, & Roth, 2002). 

Promote self-efficacy for learning online. Research has shown that students’ self-efficacy beliefs can be 
enhanced in several ways (Schunk & Ertmer, 2000). For instance, prompt scaffolds that guide and encourage 
students to set challenging, proximal learning goals can be an effective way to (a) direct students’ attention to 
relevant task features, (b) help them select and apply appropriate strategies, and (c) encourage them to compare their 
current performance with the learning goal (Locke & Latham, 2002; Zimmerman, 2008). In a self-paced online 
course, a self-efficacy prompt might consist of a simple cue at the start of each lesson that explicitly identifies the 
knowledge and skills students are expected to acquire by the end of the lesson. Ultimately, proximal learning goals 
have been shown to encourage self-monitoring and self-evaluation, behaviors that strengthen self-efficacy and 
sustain motivation (Bandura, 1997; Schunk et ah, 2008; Schunk & Zimmerman, 2008). 

Address boredom and frustration. Findings from this study suggest that negative achievement emotions are 
generally associated with less metacognitive activity, decreased satisfaction, and reduced continuing motivation. In 
practical terms, implications can be based on the basic assumption that learning will be improved when negative 
emotions are minimized and positive emotions are maximized (O’Regan, 2003; Pekrun, 2006). Of course, the 
unanticipated positive relationship between frustration and metacognition found here suggests that the links between 
emotions and self-regulatory behaviors may be more complex than this simplistic statement implies. Nevertheless, it 
seems that instructional designers and policy makers would do well to address those areas of course design and 
delivery where negative emotions are likely to be directed: namely, the learning task and the technology (Wosnitza 
& Volet, 2005). In the end, however, the most important step toward improving self-paced online learning may be 
for instructional designers and students alike to simply become aware of the close relationships between motivation, 
emotion, and self-regulated learning, as highlighted by the findings of the present study. 

Limitations and Future Directions 

The results of this study were strictly correlational; therefore, one cannot infer causality from the observed 
relationships. Although the findings suggest fairly robust relations between the measured variables, the direction of 
influence between the variables is a bit ambiguous, and thus more controlled studies that utilize longitudinal or 
experimental designs are needed before definitive pathways can be established. Furthermore, additional studies that 
employ experimental or quasi-experimental designs are needed to determine if the implications suggested in the 
previous section are capable of positively influencing self-regulated learning and online success. 

Another important limitation was the use of self-reports to examine students’ beliefs, emotions, and 
behaviors. Like all self-report measures, the survey used in this study has reliability and validity limitations (e.g., 
social desirability bias and response sets; Thorndike, 2005). Particularly important in the present study, however, are 
the threats to construct validity that are inherent to measuring emotions and behaviors with a questionnaire. With 
respect to emotions, some have argued that “this style of measurement cannot provide direct access to emotions as 
they unfold during the learning process” (Wosnitza & Volet, 2005, p. 452). Likewise, in terms of measuring how 
students actually employ self-regulated learning behaviors and how those tactics are strategically changed during 
learning, some scholars have maintained that self-reports are severely limited (Boekaerts & Cascallar, 2006; 

Hadwin, Nesbit, Jamieson-Noel, Code, & Winne, 2007). Altogether, these measurement limitations suggest that 
future work should include alternative techniques for assessing the various aspects of academic self-regulation. 
Additionally, it seems that no single instrument is sufficient to measure self-regulated learning; instead, a 
combination of assessment tools is probably necessary to capture what students actually think, feel, and do in 
various academic contexts (Boekaerts & Cascallar, 2006). 
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Conclusion 



Considered together, results from this study provide some insight into the complex relations between 
personal and behavioral influences on self-regulated learning and overall academic success in an online course. 
Notwithstanding methodological limitations, these findings largely support the existing literature on self-regulation 
in traditional, classroom-based contexts (e.g., Pekrun et ah, 2002; Pintrich, 1999; Pintrich et al., 1993; Zusho et al., 
2003). Furthermore, these results offer important theoretical and empirical extensions of academic self-regulation by 
illustrating that several processes and interrelations are equally robust in self-paced online learning situations. In 
particular, the findings reported here are consistent with the social cognitive notion that students’ motivational 
beliefs and negative achievement emotions are powerful predictors of their self-regulated learning behaviors, 
satisfaction, and continuing motivation. These results are noteworthy because they further inform our understanding 
of both online learning and academic self-regulation. Moreover, these findings shed some light on the links between 
achievement emotions and several adaptive outcomes — relationships that have been largely neglected in educational 
research (Linnenbrink & Pintrich, 2002; Picard et al., 2004). 

Results from this study also suggest that social cognitive theories of self-regulation provide a useful 
framework for understanding student success in online situations. Accordingly, future studies should continue to 
apply such multidimensional models of learning to further inform our understanding of the complex relations 
between students’ thoughts, feelings, and actions during online learning. Additionally, future work should examine 
whether online instructional interventions designed from a self-regulated learning perspective can actually improve 
learning and performance in these highly independent contexts. Ultimately, pursuing such work has the potential to 
move the field forward by providing additional guidance for the theory, research, and practice of online learning. 
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“The usefulness of what is learned is generally a greater influence on adults’ motivation to learn than 

its intellectual value ” (Wlodkowski, 2008, p. 98) 

Introduction 

Every year, thousands of adults return to school for personal fulfillment, to improve their job skills, or to 
begin a new career (U.S. Department of Education. 2007). Many of them will chose to pursue degrees, certificates, 
or professional development via online learning because of their convenience and/or because of their cost saving 
benefits (Campbell, 2008; Eield, 2008; Young, 2008). According to the Sloan Consortium report. Making the 
Grade: Online Education in the United States 2006 (Allen & Seaman, 2006), population trends in online enrollment 
are similar to on campus enrollment. This means that many of the adults returning to higher education will find 
themselves sharing online “classroom” space with “traditional” (entered college after high school, financially 
dependent, works part-time or not at all) college students who are taking some or all of their courses online. 

Thinking about the motivational design of instruction, educators and instructional designers of online learning 
environments should consider the differences between these two populations. 

Consider Student A and Student B. Student A, the “traditional” student, woke up at 11am for her noon on 
campus biology lecture, watched afternoon soaps on television while logging into her online university extension 
course in social marketing course to take notes for a reflection paper she’d write tomorrow, dropped into the 
cafeteria for dinner, went out for drinks with friends to watch the basketball game, and then went to bed at 1 1:30pm. 
Student B, the “non-traditional” student, woke up at 4:30am to get ready for work, dropped his kids off at childcare, 
worked at 9 hour day at his advertising firm, picked up his kids from childcare, made dinner, helped the kids with 
homework, put them to bed, logged into his university extension course in social marketing to began writing a 
reflection paper, and then went to bed at 1 1:30pm after falling asleep twice on the computer keyboard. 

Even though Student A and Student B are taking the same online course, the scenarios above depict at least 
four primary differences: personal time, responsibilities, priorities, and experiences. Unlike their younger 
classmates, an adult learner is motivated to learn that which satisfies needs and interests, is life-centered, and allows 
them to be self-directed (Knowles, 2005). In addition to these differences, foundational adult education researcher 
Lindeman pointed out that the individual differences between people increases with age (1945). This means not 
only are adult learners likely to be different than their younger counterparts, they are also more likely to be different 
from each other. Reflecting on Ratey’s (2001, p. 247) definition of motivation, a process that “determines how 
much energy and attention the brain and body assign to a given stimulus,” it could also be concluded that what 
motivates each of these students to learn is different, too. 

How can online educators and instructional designers motivationally design instruction so it is relevant to a 
diverse population of learners? One may point to the variety of instructional techniques that allow for the 
differentiation, but in cases where the skills being taught or the information being shared are necessary for all 
learners to know, how do educators address the distinctive beliefs and characteristics of their learners without 
developing a personalized lesson for every student? While it may not be logistically possible to differentiate all 
instruction, tailored lesson introductions may offer a means to efficiently and effectively motivate learners to learn. 

Literature Review 
What is tailoring? 

Tailoring is a design technique that incorporates formally assessed learner characteristics into the message 
design, thus making the message personally relevant. Strecher and Kreuter (1999) explain the rationale for using 
tailoring as a process. By tailoring materials, unessential information included to make the learning task more 
attractive is eliminated; what remains is more personally relevant. When information is relevant, it is more likely to 
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be thoughtfully processed, and thus, more successful in guiding a person to make a suggested behavior change 
(Kreuter and Wray, 2003; Petty and Cacioppo, 1981; Strecher and Kreuter, 1999). This, in part, is due to the fact 
that relevancy leads to an optimistic task-appraisal, consequently increasing learners' commitment to furnish effort 
(Boekaerts, 1992, 1996). 

In health education studies, where tailoring has more commonly been researched, tailored instruction 
materials have shown to be more effective in generating interest, increasing knowledge, and eliciting behavior 
change than non-tailored materials (Kreuter, Farrell, Olevitch, and Brennan, 2000). For example, Prokorhov and 
colleagues (2007) found that tailored, smoking cessation software was rated as more relevant and participants 
exposed to the tailored software were more likely to progress towards abstinence. In another study. Rimer and 
colleagues (2002) found that participants who were sent tailored breast cancer and mammography education 
materials had significantly greater knowledge and more accurate risk perceptions than participants who were sent 
non-tailored, materials. Similar results were uncovered by Latimer, Katulak, Mowad, and Salovey (2005) using 
tailored cancer prevention messages; participants were more likely to change lifestyle behaviors and request early 
cancer detection tests. Finally, in a dental education study, Updegraff, Sherman, Luyster, and Mann (2007) found 
that participants were more likely to process information and have a positive attitude towards flossing after exposure 
to tailored educational messages. Additionally, they found that the tailored messages had positive impact on 
subsequent flossing behavior. 

The case for relevancy 

What once could only be theorized, the optimistic task appraisal and ease of processing relevant 
information can now be explained by neuroscientific research. Zull (2002), in The Art of Changing the Brain, 
makes a strong case for building relevance and doing so early on instruction because of the way the brain processes 
new information. Simplified and summarized, the learning cycle, from a neuroscientific standpoint, is as follows: 

1 . The brain receives information from the world by way of one or more of the five senses. 

2. The brain attempts to make connections between the information and previous experiences 

3. The brain uses said found connection to create an image, derive meaning, or make a plan. 

4. The brain actively tests the plans and ideas through physical action (ex. speech, writing, or non-verbal 
movement. 

In looking at these four steps, it is easy to predict what may happen if a learner cannot make a connection to 
previous experiences or existing knowledge . . . learning may fail to take place. If a learner cannot connect to the 
information being presented, time and energy could be wasted seeking a connection, thus missing out on the 
instruction that is unfolding. Worse yet, the learner may make the choice not to learn. The latter outcome is even 
more likely to occur with adult learners; adult learners, according to Bee and Bjorkland (2004), are less likely to 
consider information they find irrelevant. Zull (2001) explains that as we age and the more repeated connections we 
make to existing neural networks, the more difficult it is to break physically apart those connections and create new 
ones. Hence the phrase: “You can’t teach an old dog new tricks.” Old dogs, however, CAN learn. It’s simply a 
matter of educators facilitating unique learner connections, from the beginning (i.e. the introduction), to things that 
they already know and reinforcing the new connection. 

This neuroscientific information into the role of relevancy could help to explain the results of the health 
education studies cited above. Learners who were exposed to tailored materials were more likely to demonstrate an 
increase in knowledge and change in behavior than those exposed to non-tailored materials. In tailoring the 
materials, learners were able to build upon existing neural networks instead having to build new ones. 

Is relevance, however, reason enough for a learner to pay heed to instruction? Perhaps it is not. 

Wlodkowski (2008) purports, “The usefulness of what is learned is generally a greater influence on adults’ 
motivation to learn than its intellectual value” (p. 98). This means that the novelty of learning is simply not enough; 
instruction, in order to be effective, must demonstrate that what is learned will, in some way, satisfy learners’ needs. 
How can an educator tailor instruction so that it not only appears relevant to each learner, but useful? What 
“formula” can educators follow so that are not guessing how best to do this? Behavioral construct tailoring may 
offer the prescriptive approach. 

Behavioral Construct Tailoring 

While tailoring is a means to increase relevancy, behavioral construct tailoring makes the content 
strategically relevant. Behavioral construct tailoring, according to Kreuter, Oswald, Bull, and Clark (2000), is when 
established behavioral construct theories are used to form the structural content of a message. Such theories could 
include, but are not limited to, Bandura’s social cognitive theory (1986), Prochaska and DiClemente’s stages-of- 
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change theoretical model (1983), Ajzen’s theory of reasoned action (1992), or Deci and Ryan’s self-determination 
theory (1985). In this regard, behavioral construct tailoring is a strategic means to stimulate motivation. 

To design a behavioral construct tailored message, an educator must first identify the behavior to be 
introduced, changed, or reinforced. In this study, for example, the selected behavior was learning website evaluation 
skills. While other learning tasks could have been selected, this one was chosen because it was already an existing 
part of the study participants’ regular curriculum, thus the research could take place in a naturalistic setting. 
Additionally, considering 75% of people rely only on common sense or a casual protocol when validating online 
information (Fox and Rainie, 2002), the task could be regarded as an undervalued skill in need of motivational, 
instructional enhancement. 

Selecting the Behavioral Construct 

After the behavior has been identified, the next step is to select a theory or model that explains it. This 
theory will provide structure to the instructional message and determine the type of audience data to collect. For this 
study, that theory was the Extended Parallel Process Model (EPPM) (Witte, 1994, 1998). 

The EPPM is a fear appeal theory that suggests messages act as an external stimulus to initiate two 
different cognitive appraisals, threat and efficacy. The threat appraisal pertains to perceived susceptibility to and 
perceived severity of a threat. The threat, in an instructional environment, could be negative consequences 
stemming from not performing the learning task. The efficacy appraisal relates to the perceived ability to perform 
the learning task and its ability to reduce or eliminate negative consequences. Based on the appraisal, one of three 
response states surface: low/no response, danger-control, or fear-control (Witte, Meyer, and Martell, 2001). These 
three states would come to be addressed in the tailored introductions. 

Although another theory could have been chosen, the EPPM was selected for two reasons. First, its 
persuasive message design roots trace back to the fear appeal studies conducted by Hovland, Janis, and Kelley 
during World War II (Petty, Barden, and Wheeler, 2002; Witte et ak, 2001), as well as persuasive message studies 
conducted by Leventhal, Watts, and Pagano (1967) and Rogers (1975). Although fear need not lie at the root of 
education, awareness of consequences can be a motivational instrument. Second, an audience assessment tool based 
on the model already exists. 

Assessing Learner Characteristics 

Once a theory has been identified, the next step is to gather learner characteristic data in relationship to that 
theory. The Risk Behavior Diagnosis (RBD) scale template (Witte, Cameron, McKeon, and Berkowitz, 1996), a 12- 
item template theoretically grounded in the EPPM, was selected for this study (see Method). This template permits 
researchers to drop an observed risk and a suggested response into the context of the assessment. The risk refers to 
consequences associated with not practicing a suggested behavior. Results of the RBD indicate whether one is in a 
state of danger control, fear control, or low/no control. Knowing a one’s state, an educator can deliver a message 
that motivates the individual to practice the suggested response. 

Tailoring in Other Fields 

Although many concepts thus far have pointed towards health and safety, it is necessary to show how they 
apply to other fields. Eor example, consider the high school student who drops her math course because “she’ll 
never need it.’’ Or the college student who skips his “boring” business writing class, but needs to write a cover letter 
for a job. Or the records clerk who skips a mandatory online workshop about client confidentiality, claiming that it 
is irrelevant to her division. If these individuals perceived a severe and likely negative outcome resulting from non- 
performance, and if they believed they could avoid those outcomes by performing the suggested learning task, they 
would be more likely to engage. Such is the case made by the EPPM (Witte, 1994, 1998) and assessed by the RBD 
(Witte, Cameron, McKeon, and Berkowitz, 1996). 

Purpose 

Behavioral construct tailoring provides a prescriptive means to improve motivation to learn, but additional 
research in other fields is needed. The conceptual basis for this study was that tailoring just a lesson’s introduction 
would be enough to generate greater motivation, as measured by pre and post lesson task attraction and task 
relevancy, to perform a learning task. Additionally, since it was expected that learners exposed to the tailored lesson 
introduction would make a quicker connection to the lesson, greater perception of task competence, as measured 
after the lesson, was anticipated. The hypotheses were: 

1 . Participants exposed to behavioral construct tailored lesson introductions will report greater task attraction and 
relevance prior to instruction than participants exposed to the standard lesson introduction. 
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2. Participants exposed to behavioral construct tailored lesson introductions will report greater task attraction and 
relevance after instruction than participants exposed to the standard lesson introduction. 

3. Participants exposed to behavioral construct tailored lesson introductions will report greater perceived task 
competence after instruction than participants exposed to the standard lesson introduction. 

Method 

This study blended experimental research with classroom-based research. A controlled trial was used to 
compare an experimental group to a control group on the basis of exposure or non-exposure to a behavioral 
construct tailored lesson introduction prior to instruction. Instruction was an online tutorial available through Xavier 
University, Cincinnati, Ohio entitled “Evaluating Websites” which teaches learners how to locate reliable and 
relevant internet resources. Participants accessed this tutorial during their regularly scheduled class time. 

Participants 

With Institutional Review Board approval, participants were recruited from six sections of an education 
technology course at a Midwestern university during the fall 2006 semester. Of these six sections, five instructors 
agreed to let their students participate and allow time for the study. All present students participated. 

Procedures 

The researcher spoke with the instructors about the goals of the study and gave them equal numbers of 
experimental and control group packets containing the assessments and directions. Packets were alternately stacked 
to automatically randomize participant assignment to experimental and control groups. Students completed the 
activity in class and were awarded extra credit points for participation. The first step of the directions asked students 
to complete and self-score a RBD about website evaluation skills developed for this study (see Figure 1 ). Based on 
their score, experimental group participants, unbeknownst to them, were directed to a tailored tutorial introduction 
web page matching their response state; regardless of their score; control participants, on the other hand, were 
directed to a web page linking to the existing, non-tailored introduction which states: “This tutorial will help you 
locate reliable and authoritative Web resources that are appropriate for your research project” (Xavier University, 
n.d., f 1). All participants then proceeded to the tutorial. 

Intervention 

Three behavioral construct tailored lesson introductions, one for each of the EPPM response states, were 
developed to make the content of strategically relevant and more attractive to the participant. Because a lesson’s 
introduction is one of the first places to capture learners’ attention, the introduction was regarded as the lesson 
component most in need of treatment. Structurally, each introduction was designed to address perceptions of threat 
and efficacy in relationship to the learning task, learning how to evaluate websites. Results from a formative 
evaluation with a similar audience guided decisions about the message type (realistic, humorous, etc.) and content 
(examples given and primary referents). Each introduction was a web page hyperlinked to the tutorial. The fear- 
control introduction recognized the threat and increase efficacy perceptions (see Figure 2). The danger-control 
introduction reinforced existing threat and efficacy perceptions (see Figure 3). Finally, the low/no control 
introduction highlighted the seriousness of and susceptibility to the threat (see Figure 4). The threat, in all cases, 
was negative consequences stemming from not performing the learning task. 

Because the danger-control state is when one is most likely to take action (Witte, 1994, 1998), the 
introductions were designed to move participants in that direction by way of increasing perceived efficacy and threat 
or reinforce it if they were already there. Even though consequences may not be health or safety related, examples 
given in the introduction were health and safety related to better match the selected behavioral construct and 
assessment tool. 

Assessment Materials 

1. The Risk Behavior Diagnosis Scale: The RBD scale is used to predict responses to a given threat. Each 
item in the RBD is based on a 7-point scale ranging from “1 -strongly disagree” to “7-strongly agree.” To determine 
one’s state, users rate agreement, on a 7-point scale ranging from “1 — strongly disagree” to “7-strongly agree,” with 
12 statements split between two categories, threat and efficacy. To calculate a score, the sum of the threat ratings is 
subtracted from the efficacy ratings. This number is called the discriminating value. Positive values indicate danger 
control, negative values indicate fear control, and low values (1 or 2) indicate low/no threat control (Witte et ak, 
2006). A reliability test on the RBD constructed for this study revealed a Cronbach’s alpha of .752. 
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Survey Strongly Strongly 

Disagree Agree 

1. Learning website evaluation criteria is an effective way to prevent someone from 1 2 3 4 5 

not recognizing poor quality health information found on the Internet. 

2. Learning website evaluation criteria would work to keep someone from not 1 2 3 4 5 

recognizing poor quality health information found on the Internet. 

3. If I learned website evaluation criteria, I would be less likely to not recognize 1 2 3 4 5 

poor quality health information found on the Internet. 

4. If given the opportunity, I would be able to learn how to evaluate websites. 1 2 3 4 5 

5. 1 have the time to learn how to evaluate websites. 

6. 1 could easily learn how to evaluate websites. 1 2 3 4 5 

1 2 3 4 5 



Strongly Strongly 

Disagree Agree 

7. 1 believe that not recognizing poor quality health information from the Internet 1 2 3 4 5 

could be dangerous. 

8. 1 believe that not recognizing poor quality health information from the Internet 1 2 3 4 5 

could have negative outcomes. 

9. 1 believe that not recognizing poor quality health information from the Internet 1 2 3 4 5 

could be extremely harmful. 

10. It is likely that I would not recognize poor quality health information from the 1 2 3 4 5 

Internet. 

11.1 am at risk for not recognizing poor quality health information on the 1 2 3 4 5 

Internet. 

12. It is possible that I might not recognize poor quality health information from the 1 2 3 4 5 

Internet. 



Figure 4 Risk Behavior Diagnosis Scale - Website Evaluation 

(Format adapted from Witte, K., Cameron, K. A., McKeon, J. & Berkowitz, J. (1996). Predicting risk behaviors: 
Development and validation of a diagnostic scale. Journal of Health Communication, 1, 317-341. 
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2. Motivation Survey: A modified version of Boekaerts’ (2002) Online Motivation Questionnaire (OMQ) 
was constructed to measure motivation, in terms of task attractiveness and relevancy, before and after the tutorial, as 
well as performance attributions, in terms of competence. Designed to “drop” a learning task into its template-like 
format, the OMQ was an ideal choice. Multiple-choice questions were reworded so users could respond on a 
“Strongly Agree” to “Strongly Disagree,” 5-point Likert scale. 

Motivation Surveys Part 1 and 2 each consisted of six questions related to two motivational subscales, task 
relevance and task relevancy (see Figures 5 and 6). Part 1 was completed after the introduction, but before the 
tutorial; Part 2 was completed after the tutorial. Motivation Survey Part 3 consisted of three questions related to their 
competence as a performance attribution and was completed after Part 2 (see Figure 7). Cronbach’s alpha for 
Motivation Survey Parts 1, 2, and 3 were .906, .928, and .777, respectively. 

Results 

All 98 students present on the day of the study participated. Students’ ages ranged between 18-52 years. 
Ninety-one percent of the participants were under the age of 28, eighty percent were under the age of 23. Eighty- 
nine students (91%) were female, eight were male (8%), and one (1%) did not report. 

Hypothesis 1 

Data on two different subscales, task attractiveness and task relevancy, were analyzed for variances in 
initial student motivation. An independent-measures t-test was used to determine the effects of the treatment on 
motivation by comparing mean subscale scores of the control group (N=50) to mean subscale scores of the 
experimental group (N=48). After a Bonferroni correction was applied, alpha was determined to be .04. 

An analysis of results showed that the experimental group reported significantly higher levels of motivation 
than the control group for the subscale “task attractiveness,” f(96) = 3.425, p < .001; d = .737) (see Table 1). The 
mean “task attractiveness” rating was 3.23 (SD= 0.799) for the experimental group and 2.64 (SD= 0.898) for the 
control group (See Table 2). Results also revealed a positive trend toward higher levels of motivation for the 
experimental group, for the subscale “task relevancy,” f(96) = 2.212, p < .Q5\d = .450. 
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susceptibility 


Altnougn the internet is a seemingly endless 'Source 
of health information, it's not always a goldmine. 
According to one researcher, less than 45% of medical 








websites are reliable. This means that you could be 
looking up info on the funky rash you keep getting or the 
new nutrition =:ijpplpmpnt you'vp hp^ takir^] to losp 




threat 






response-efficacy 


weight and only every other website is golhg.^ be 
accurate. 
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Figure 2 : Fear-Control Introduction 
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Although the internet is a seemingly endless 
source of health information, it can also be a land- 
mine. According to one researcher, less tttafT?5%’"of^ 
medical websites are reliable. Unknowingly buying 
into a poor quality website could lead to some pretty 
scary outcomes! For example . . . you find a 
website about the weird rash you keep gettiimjHidit 
says it's no big deal. You breathe a sighpffdief but 
in actuality you are in the late stag^S"bfa super 
serious disease. A couple of situations like these 
could not only lead to emotional distress, but also 
loss of health! Are you confident that you can 
recognize the 55% of medical websites that are 
unreliable? 

You need to protect yourself and now is the 
time to start doing it. This quick tutorial will show 
you how to distinguish the diffefenteHoetweejiJii^^ 
and low quality websites, thus making you a wiser 
and healthier consumer. 



FIgureB : Danger-Control Introduction. 
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Although the internet is a seemingly endless 
source of health information, it can also__^ 
mine. According to one researcher, less^an 
45% of medical websites are reliable. This rrueans 
that you could be researching nutrition 
supplements for your dad or looking up^info about, 
the weird rash you keep getting ^rra only ev^ 
other site is going to be accurate. Whap^-^u 
bought your dad one of the supplep^efTts 
advertised and an ingredient thewebsite forgot to 
list interferes with his blood pressure meprcation? 

Or the website you found about your rich says 
it's "nothing," when in fact it's life-threatening? 

Are you absolutely confident that you can 
recognize the 55% of medical websites that are 
unreliable? 



If you are going to use the internet to look 
up health information, you've got to protect 
yourself. The following tutorial will quickly 
introduce you to website evaluation criter^^--th^t 
will make you make you a wiser and healthier, 
more efficient consumer. 




response-efficacy 



Figure 4 : Low/No Threat Control Introduction. 
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Questionnaire Part 1 


Strongly 




Strongly 






Disagree 




Agree 




1 . 1 am in the mood to learning how to evaluate websites { attractiveness } 


1 


2 


3 


4 


5 


2. 1 am interested in learning how to evaluate websites 


1 


2 


3 


4 


5 


{ attractiveness} 

3. Learning how to evaluate websites is intriguing, {attractiveness} 


1 


2 


3 


4 


5 


4. Learning how to evaluate websites is useful to me {relevance} 


1 


2 


3 


4 


5 


5. Learning how to evaluate websites is personally relevant to me {relevance} 


1 


2 


3 


4 


5 


6. Learning how to evaluate websites is helpful to me {relevance} 


1 


2 


3 


4 


5 



Figure 5 Motivation Survey Part 1 
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Strongly 




Strongly 


1. 1 was in the mood to learning how to evaluate websites {attractiveness} 


Disagree 
1 2 


3 


Agree 

4 
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2. 1 was interested in learning how to evaluate websites { attractiveness } 


1 


2 


3 


4 


5 


3. Learning how to evaluate websites was intriguing.} attractiveness} 


1 


2 


3 


4 


5 


4. Learning how to evaluate websites was useful to me. {relevance} 


1 


2 


3 


4 


5 


5. Learning how to evaluate websites was personally relevant to me 


1 


2 


3 


4 


5 


{relevance} 

6. Learning how to evaluate websites was helpful to me. {relevance} 


1 


2 


3 


4 


5 



Figure 6 Motivation Survey Part 2 



Questionnaire Part 3 


Strongly 




Strongly 


With regards to my performance and the tutorial . . . 


Disagree 




Agree 


1)1 am good at this type of task 


1 2 


3 


4 5 


2) 1 already know a lot about website evaluation criteria 


1 2 


3 


4 5 


3) 1 knew the best way to go about learning it 


1 2 


3 


4 5 



Figure 7 Motivation Survey Part 3 
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Table 1; Effects of Treatment on Pre-Instruction Attraction and Relevancy 







P 


MD SE 


95% Confidence 
Interval 










Lower 


Upper 


Attraction 


Equal 


.001 


.589 .172 


.248 


.931 


■ In the mood 

■ Enthusiastic 

■ Appealing 


variances 

assumed 










Relevance 


Equal 


.036 


.358 .169 


.022 


.692 


■ Useful 

■ Personally relevant 

■ Helpful 


variances 

assumed 










Table 2: Pre-instruction Mean Relevancv and Attraction Scores 


















Motivation subscale 




Message 

Type 


Mean 


SD 


SE 


Attraction 




Tailored 


3.229 


.799 


.115 


■ In the mood 

■ Enthusiastic 

■ Appealing 




Standard 


2.640 


.898 


.127 


Relevance 




Tailored 


3.937 


.795 


.115 


■ Useful 

■ Personally relevant 

■ Helpful 




Standard 


3.580 


.870 


.123 
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Hypothesis 2 

To address the second hypothesis, data were analyzed for variances in post-instruction student motivation, 
specifically task attractiveness and task relevancy. An independent-measures t-test was used to determine the 
effects of the treatment on motivation by comparing mean subscale scores of the control group (N=47) to mean 
subscale scores of the experimental group (N=49). After a Bonferroni correction was applied, alpha was determined 
to be .04. 

An analysis of results showed that the experimental group reported significantly higher levels of motivation 
than the control group for the subscale task attractiveness, f(96) = 3.016, p < .001; d = .586) (see Table 3). The 
mean task attractiveness rating was 3.440 (SD= 0.820) for the experimental group and 2.925 (SD= 0.851) for the 
control group (See Table 4). Unlike before instruction, there was not a trend toward increased task relevancy. 

Table 3: Effects of Treatment on Post-instruction Attraction and Relevancy 



p MD SE 95% Confidence 

Interval 



Lower Upper 



Attraction 

■ In the mood 

■ Enthusiastic 

■ Appealing 

Relevance 

■ Useful 

■ Personally relevant 

■ Helpful 



Equal .003 
variances 
assumed 



Equal .121 
variances 
assumed 



.514 .170 .176 



.308 .197 .083 



.853 



.700 



Table 4: Post-instruction Mean Attraction and Relevancy Scores 



Motivation subscale 


Message Type 


Mean 


SD 


SE 


Attraction 


Tailored 


3.440 


.820 


.120 


■ In the mood 

■ Enthusiastic 

■ Appealing 


Standard 


2.925 


.851 


.122 


Relevance 


Tailored 


3.731 


.929 


.136 


■ Useful 

■ Personally relevant 

■ Helpful 


Standard 


3.422 


.999 


.143 
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Hypothesis 3 

To address the third hypothesis, data were analyzed for variances in post-instruction performance 
attributions in terms of competence. An independent-measures t-test was used to determine the effects of the 
treatment on motivation by comparing mean subscale scores of the control group (N=47) to mean subscale scores of 
the experimental group (N=49). After a Bonferroni correction was applied, alpha was determined to be .04. 

An analysis of results showed that the experimental group reported significantly higher levels for the 
performance attribution competence than the control group, f(94) = 2.087, p < .040; d = .426. (See Table 5). The 
mean competence rating was 3.702 (SD= 0.720) for the experimental group and 3.381 (SD= 0.785) for the control 
group (See Table 6). 



Table 5: Effects of Treatment on Post-Instruction Task Competence 



Motivation subscale 


Message Type 


Mean 


SD 


SE 


■ Good at this type of 


Tailored 


3.702 


.720 


.105 


task 

■ Already knew a lot 
about this task 

■ Knew the best way to 


Standard 


3.381 


.785 


.112 


learn it 











Table 6: Post-Instruction Mean Task Competence 






P 


MD SE 95% Confidence 

Interval 


Lower Upper 


■ Good at this type of 
task 

■ Already knew a lot 


Equal 

variances 

assumed 


.040 


.322 .154 .0157 .627 



about this task 
■ Knew the best way to 
learn it 



Limitations 

Generalizations could be hindered by three major limitations. First, the RBD may have cued learners to 
consider their ideas about the learning task when they might not otherwise have, thus possibly eliciting an 
interaction effect of testing and selection. Second, the subject pool was limited to students enrolled in an educational 
technology course, thus possibly eliciting a selection bias. Third, the sample was gender homogenous; the majority 
(91%) was women. Although limitations do exist, rather than detracting from the utility of the findings, they should 
prompt others to conduct more research into tailoring. 

Discussion 

If one simply considers tailoring a good marketing technique, tailored instruction makes sense. It makes 
even more sense when designing online learning environments where there may be a limited exchange of feedback 
between learners and their instructors, and where learners may be required to self regulate their own learning. If the 
instructor cannot be there to help build the relevancy needed to ignite the learning process (ZuII, 2002), that 
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relevancy needs to be built into the online instructional materials. That relevancy is particularly important when 
working with adults; as indicated earlier, adults are less likely to consider information that they find irrelevant (Bee 
and Bjorkland, (2004). Considering the scenario of Student A and B presented in the introduction, it’s easy to 
understand that designing learning environments requires a more tailored approach. Results of this study indicated 
that behavioral construct tailored lesson introductions are the kind of design feature that may help educators of 
electronic based learning environments feasibly reach a wider audience. 

The intention of the first hypothesis was to see if a behavioral construct tailored lesson introduction could 
propagate higher, self-reported levels of motivation toward an online tutorial without any motivational coaching 
from an instructor. Results showed the experimental group reporting significantly higher levels of motivation for the 
subscale task attractiveness. In other words, the presence of a tailored introduction led this group to perceive the 
learning task as more attractive, even before commencement. The attractiveness subscale was made up of the 
components: in the mood, enthusiastic, and appealing. The effect size indicates that the tailored introduction had a 
fairly good impact on task attractiveness. There was also a trend towards higher levels of task relevancy. 

Though not hypothesized, a strong correlation was also found to exist between task relevancy and task 
attractiveness both before (r =.682; p< .001) and after instruction (r =.830; p< .001). This relationship supports 
Boekaerts’ (1992, 1999) finding that relevancy leads to an optimistic task-appraisal (i.e. task attractiveness). The 
benefit of an optimistic task appraisal is that it may lead to improved intentions to learn (i.e. effort). In a math 
homework study, for example, Boekaerts (2002) found a significant relationship between students’ appraisals of a 
learning task and their commitment to perform that task. Similarly, in an English as a Foreign Language (EEL) 
course study, Bemaus and Gerdner (2008) found that attitudes toward the learning situation predicted motivation to 
learn English. Additionally, they found that motivation was a positive predictor of English achievement. Thus, 
gains in relevancy and/or task attractiveness should be regarded as positive outcomes. 

The intention of the second hypothesis was to observe the sustained impact of a behavioral construct 
tailored lesson introduction on task attraction and task relevancy. Results revealed, again, greater perceived task 
attractiveness for experimental group participants. There was no positive trend, however, toward higher levels of 
perceived of task relevancy. Conversely, a repeated measure, one-way ANOVA revealed an overall, significant 
negative change for task relevancy. In analyzing the interaction effects, however, the lesson introduction (the 
intervention) was not the cause. This means the change in perceived task relevancy was elicited by another factor or 
combination of factors. If having a positive attitude towards a task leads increased likelihood to perform a similar 
task again (Bandura, 1977, 1986, 1998; Weiner, 1986) it would be worthwhile to identify the cause of this change so 
as to promote enhancers and lessen detractors. 

The intention of the third hypothesis was to observe the impact of a behavioral construct tailored lesson 
introduction on perceived competence. Results showed that participants exposed to the tailored introductions were 
more likely to attribute their performance to competence. One reason for the increased competence could be the 
behavioral construct used to guide the design of the tailored lesson introductions, the extended parallel process 
model (EPPM). One of the EPPM factors assessed and imbedded into the introductions was self-efficacy. It is 
possible imbedding a layer of self-efficacy into the introductions helped to foster greater perceived task competence. 
Although the effect size was moderate, the outcome is a positive one. Competence has been shown to correlate with 
achievement (Bouffard, Marcoux, Vezeau, and Bordeleau, 2003; Gonida, Kiosseoglou, and Leondari, 2006 )and to 
positively impact task persistence (Bandura, 1998). While improved academic achievement is definitely a desirable 
outcome, so, too, is task persistence, particularly in online learning environments where learners must self-motivate 
to perform (Dillon & Greene, 2003). 

Future Research 

The concept of tailored instruction falls right into the growing trend towards personalization. The 2007 
Horizon Report discusses personalization in terms of user-created content where the learner is no longer just an 
audience, but a participant in the instructional design (New Media Consortium, 3). Tailoring offers limitless 
opportunities for new research. Such possibilities include: 

1. Tailoring as foundation for message design - In this study, the content of a lesson introduction was tailored 
to learners’ unique perceptions about a learning task. Future research into tailoring, as an instructional 
message design technique for online or other electronic learning environments, should be investigated. 

2. Tailoring other parts of instruction - In this study, only the lesson introduction was tailored. Future studies 
could investigate the impact of tailoring other or more sections. 

3. Learning Environments - In this study, an online tutorial was motivationally adapted. Future studies could 
investigate tailoring other types of electronic learning environments such as software and digital textbooks. 
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4. Behavioral constructs - In this study, tailoring was based on a risk message behavioral construct. Future 
studies could investigate the use of other behavioral constructs. 

5. Academic Achievement - This study primarily focused on motivation. Future studies should also 
investigate the impact of tailored introductions on valid, reliable cognitive assessments. 

Research studies like these would help to expand the use of tailoring to other academic areas and other aspects 
of instructional design. 

Conclusions 

Wlodkoswki (2008)shares that “the value in coming to better understand motivations is so we can use 
attention, concentration, imagination, passion, and other processes to pursue goals” (p. 3). Without doing so, leaves 
only trial and error. In taking small steps to strategically assess learner characteristics to build those known 
characteristics into the early part of instruction, one better stands to motivate his or her audience. 

Knowing that a tailored lesson introduction, alone, was sufficient to elicit such results tells educators and 
instructional designers that they might not have to go to extremes to personalize instruction. Because the study 
findings indicated that tailoring was associated with greater motivation and perceived competence, further research 
into its use when designing online courses, software, and other electronic forms of instruction is warranted. Further 
research into the value of its use when addressing socially, economically, and experientially diverse learner 
populations is also warranted. The key finding to be taken from this study is that learners are unique and that 
tailoring affords instructional designers and educators a motivational design tool to address those unique qualities. In 
an age where there are so many competing messages and bits of information with which to contend, by taking small 
measures to meaningfully design instruction, one can better maximize the true potential for knowledge to be gained. 
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An Exploratory Study of Kids as Educational Computer Game Designers 

Ahmet Baytak, Susan M. Land, Sunghyun Park, & Brian Smith 
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Abstract 

This study investigated how children designed computer games as artifacts that reflected their 
understanding of nutrition. Ten 5* grade students were asked to design computer games with the software 
Gamemaker™ for the purpose of teaching 1®‘ graders about nutrition. The results from the case study show that 
students were able to express their personal thoughts and intentions by designing and developing realistic computer 
games in a complex programming environment. Our findings pointed to gender differences in game designs and 
other contextual factors that motivated social interaction around game design and programming strategies. 

Introduction 

In the early 1990’s, Seymour Paper! forwarded the notion of “constructionism” to suggests that learning is 
most meaningful when learners are actively engaged in building artifacts (Paper!, 1991). Constructionism is based 
theoretically on Piaget’s ideas of students as builders of their own knowledge. Piaget’s statement “that children don't 
get ideas; they make them” guides the direction of constructionism (Han & Bhattacharya, 2001). Constructionism is 
also tightly connected with Paper!’ s early work with Logo and its subsequent form of Lego-Logo toys. 

Paper! characterized constructionism as encompassing two interconnected processes: The first is an internal 
and an active process where students construct knowledge from their experiences of the world. The second process 
is external, reflecting the belief that students learn best by making artifacts that can be shared with others (Grant, 
2002; Kafai & Resnick, 1996; Papert, 1991). In other words, constructionism takes knowledge from an abstract 
mode to a concrete mode. Student artifacts can be anything from a poem or a webpage, to more complex artifacts 
like an origami, or a computer game. 

Papert’ s LOGO project with elementary school students to learn math is perhaps one of the best known 
projects illustrating learning-by-constructing. However, prior studies have taken place in the domains of math 
(Kafai, 1996; Harel, 1991), science (Hmelo et ak, 2000; Kafai, 2005; Brandes, 1996; Kolodner, et ak, 2003), Music 
(Gargarian, 1996), and Thai language and emotional development (Tangdhanakanond, Pitiyanuwat, & Archwamety, 
2006). 

Artifact development in constructionist environments can be facilitated in various ways. Recent work in 
“learning by designing” emphasizes constructing artifacts by computers programming computers or designing 
games (Kafai, 2005). Designing sharable artifacts that reflect students’ different styles of thinking and learning is 
key. Kafai claimed that designing artifacts by programming software helps students reformulate their understanding 
and expression of their personal ideas and feelings about not only the subject but also the artifact. Papert also sees 
programming or game making as a construction tool for personal expression and knowledge reformulation, and this 
helps students explore psychological and cultural aspects of learning (Papert, 1995). 

Background on Game Design for Constructionist Learning 

In recent years, considerable interest has been generated in the educational potential of computer games 
(Dickey, 2005). With the commercial success of virtual gaming worlds for children, such as Webkins ox NeoPets, 
computer games have become an important part of everyday life for children. Whereas most research concentrates 
on the design and effects of gaming, our research builds off the work of Papert and others (Harel, 1991; Kafai, 1 996; 
2006) to investigate the educational impact of children designing their own computer games. That is, children were 
supported to become “producers” rather than “consumers” of computer games (Kafai, 2006). 

One characteristic that is relevant to children engaging in game design is gender (Kafai, 1996). Kafai 
examined if gender played a role in children’s stereotyping styles to their designs. These styles generally originated 
from commercial video games where females are characterized as either a prize or a target. Kafai examined game 
genre, game world, game characters, interactions and narrative development. Based on this categorization, gender 
differences were apparent in some cases. For example, most of the boys designed adventure games where girls 
designed teaching games. However, little follow-up work has been done to confirm or extend these findings. 
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Constructionism also emphasizes the social nature of learning. Artifacts are developed, in part, so they can 
be publicly shared and discussed. Thus, collaborative settings are commonplace. Shaw (1996) noted that students in 
social settings engage in a cycle of development leading to external and shared social constructs. Constructionism 
mobilizes the knowledge of the community to support learning among its members (Gargarian, 1996). Hence, 
artifact development should entail exposure to activities that promote collaboration and sharing. For example, 
Kolodner and her colleagues (2003) used a “class pin-up session” to support students to collaboratively predict how 
their designed artifacts would behave, and whole-class discussion around a whiteboard to review what students 
learned form previous steps. Similarly, they used small group and community rituals to explain, justify, and prepare 
reports about what they were learning, and poster sessions to present final products with the entire study group. In 
the context of game design, Kafai (2005) also used a group discussion which helped the students share their game 
designs, ideas, and difficulties with applying the subject to games. Similarly, Brandes (1996) implemented 
ethnographic observations and small group projects into her study where students had the chance to try each others’ 
designs and leave feedback or incorporate some ideas for their own design 

The Purpose of the Study 

The purpose of this study was to explore how children designed computer games as artifacts that 
reflected their understanding of nutrition. We wanted to examine the process used by elementary school students to 
construct understanding of nutrition knowledge by designing the games. In order to accomplish this goal, we 
focused on the following research questions: 

1- What conceptions of nutrition knowledge were used or evident in the game design? 

2- What programming strategies did students use to develop their game over time? 

3- Were gender differences apparent in the design characteristics of the games? 

Method 

Participants and Context for the Study 

Ten 5* graders, 4 girls and 6 boys, participated in this study as part of their science class during a unit on 
nutrition. Prior to the game design task, students were taught basic nutrition concepts such as food groups and 
serving sizes, over a period of four class time. The main task assigned to students for our study was to design a 
computer game using Gamemaker™ software for the purpose of teaching first graders about nutrition. Two students 
in the class had prior experience using the Gamemaker™ software, but none of the remaining eight students had 
used the software before. At the end of the project, the first graders from the same school played all the games and 
provided feedback to 5*-grade game designers. Since the students were aware of the actual audience, they had to 
design games considering age of students, entertainment in the game, and nutrition knowledge of the first graders. 
Students spent approximately 8 weeks developing their games. 

The Gamemaker™ Sottvian 

All students used the Gamemaker™ software to design their computer games. This software is an object- 
oriented game design software that can be downloaded and is available free of charge. Without requiring coding 
skill, the software allows the designers to create sprites (characters), backgrounds (settings), and objects (characters 
with actions). Besides these features, designers can add sounds, or additional coding for more professional games. 
The software comes with a collection of freeware images, sounds, and sample games to help novice designers get 
started. Figure 1 shows a sample screen shot of a student’s game as it was being developed. 



40 




Figure 1: A screenshot from Erin’s Gamemaker™ platform: Adding actions to the character 




With Gamemaker™ the students start with making sprites either using the given images or making their 
own images with the image editor in Gamemaker™ . The students then turn these sprites to objects and add actions 
to each object (Figure 1). Then students design rooms (levels) and add objects to make platforms ready for playing. 
Depending on their game style, some students used only one room, but others students used more than one room. 

Procedures 

The students met for 45 minute sessions twice a week for 8 weeks. Students were first taught how to use 
the Gamemaker™ software. Similar to Hmelo and her colleagues’ (2000), students were first presented with some 
design examples and challenges in Gamemaker™ . In order for students to understand the software, they were asked 
to modify some part of the template games that came with Gamemaker™ . Two teachers, the technology teacher and 
science teacher, facilitated the students in their designs. The technology teacher, who is also the lead author, checked 
the students’ games and left feedback during the class sessions. Feedback mainly consisted of questions to students 
to prompt new ideas in the designs. He also supported the students if they had any questions on programming. The 
science teacher played students’ games and gave them guidance on implementing nutrition facts correctly. However, 
the students were not required to change their games based on teachers’ feedback. 

The study was designed to promote collaboration among students. Similar to previous studies (Kolodner et 
ah, 2003; Harel, 1991; Hmelo et ak, 2000), students were encouraged to look at the other games to not only leave 
feedback but also get ideas for their own designs. Since it was a small class, the students would typically ask the 
entire class for guidance on how to perform certain programming tasks. That collaboration was informal, in that 
students could ask for help at any time during the sessions. 

After the students designed their games, 16 first graders came into the class and tried out the games. The 
first graders played the games in pairs, with the game designer of each game (5* graders) sitting next to them. The 
first graders provided the game designers with their feedback as they played each game. All student game designers 
were given feedback by approximately 6 different pairs of F‘ grade students. Once the project ended, all ten 5* 
graders were interviewed. Interviews of approximately one hour took place on an individual basis. The interview 
began with a series of guided questions regarding students’ goals, strategies, and perceptions for the game design 
process. Emphasis was placed on what students believed they learned about nutrition, how they learned the 
programming strategies, and why they made specific game design decisions. 
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Data Sources and Design 

The research used a case study as the research methodology (Yin, 2002). This methodology is appropriate 
for investigating complex, contemporary phenomena within its authentic context. The unit of analysis for our case 
study was the entire classroom learning environment, including all students, teachers, and artifacts. Case results 
were compared against, and explained according to, previous theoretical models developed by Kafai, 1996 and 
Kafai, 2006 (Yin, 2002). 

A primary data source for this study was the participants’ computer games. Of the 10 games designed by 
the students, eight games were analyzed, as two of them were not accessible due to technical problems. The 
students’ games were saved separately by weeks, so multiple iterations of the games at various stages of 
development were examined. Most of the students started with a draft game in the first week and then decided on 
one design idea that was developed throughout the activity. 

The following data sources were collected and used in the interpretation of results: (a) students’ written 
goals for their game design: (b) interviews with the participants following the game design; (c) the participants’ 
games; and (d) classroom observations. The lead author was present for all sessions. All interviews with participants 
were transcribed and later analyzed for insights into each research question. Initially, data were examined according 
to each participant, using matrices that represented each student’s activities and verbalizations that were relevant for 
each research question. Stored iterations of the games were examined and coded for changes across time. Then, data 
were examined broadly for different examples of game design activity; similar instances were grouped according to 
major categories of the research questions. This approach allowed us to identify trends within and across participants 
for a given question (Miles & Huberman, 1994). Two researchers (the first and third authors) collaborated on the 
analyses, allowing for discussion and agreement on how the data was to be interpreted. 

Results 

The results of the study are presented according to each research question (nutrition conceptions, 
programming strategies, gender differences). 

How was nutrition knowledge used or evident in students’ game designs? 

For this question, we examined what conceptions of nutrition knowledge were evident in the students’ 
games. Based on analyses of children’s games and interview data, most students represented the nutrition concept of 
“eat healthy foods and avoid unhealthy foods’ in their game design. For instance, typically healthy foods (e.g. 
vegetables and fruits) were used as a good/positive character such as fuel, point-gaining agent, or key to the next 
level; while unhealthy foods (e.g. hamburger and desserts) were used as a negative character such as speed reducer, 
point/energy-losing agent, or game-over agent. 

Although many children simply used healthy foods as good agents vs. unhealthy foods as bad agents in 
their games like conventional game characters, the interviews revealed that a few children connected it to the 
concept of healthiness of our body. For instance, Tom’s main character moved faster when it ate or touched salads. 
Tom explained that salads were selected because they would give nutrients that our body needed. Therefore, eating 
salads would make our body stronger Tike protein’ and that’s why his main game character could move faster. 

In addition, some of the children tried to apply ‘portion size’ (eating adequate amount of each food group) 
to their game in various ways. For example. Flora tried to deliver the portion size concept indirectly by limiting the 
number of blueberries used on her game because she knew eating too much fruit is not optimal; Markus used the 
portion size concept more directly in his game: “. . .if you ate the strawberries too much or if you ate the green beans 
too often it would actually give you a “power down,” you would lose the propane that you had. . .” 

However, children’s applications of the portion size concept to their final version of the game appeared 
limited and superficial rather than complete and showing deeper understanding of the concept. This could mainly be 
due to time limitations or technical skills. For example. Flora adjusted the number of blueberries based on the 
portion size recommendation but didn’t apply correct portion size to the all the food items used on her game. The 
interview revealed that Flora was fully aware of it and wanted to apply the right portion size for all food items on the 
next version of the game if she had more time for the project. Likewise, Markus described the technical difficulties 
he experienced to apply serving size to his game: 

I: . . .Do you have an example of not eating too much? 

M: Well that’s one of the features I couldn’t find out. . .1 was going to put on a limit and it would go down 

slowly and if you ate the green beans every time it came down or every time the strawberries appeared, you 

would fall back of course 

I: So like when you were trying to put the serving size in this game too? 

M: Yes 
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I; OK, so you were aware of the serving size? How many veggies? 

M: I couldn’t find out how to put the limit on. . . 

Overall, it was observed that most of the children applied a basic nutrition concept of “fruits and vegetables are 
healthy but too many desserts is unhealthy’’ in their game design. A few children tried to apply the concept of 
portion size to their game but it was not easy for them to implement with limited time and technical skills. 

What programming strategies did students use to develop their game over time? 

To address this question, we examined the students’ planning process and strategies that they used to 
program their educational games. Although more than half of the students reported playing video games often, only 
two students had explored Gamemaker™ or similar game design programs before the project. However, by the end 
of the project, all the students were able to build functional games even in a complex object-oriented programming 
environment. Two primary strategies were used to design the games overall. One was to start from the existing 
“template” games available in the software and then to add on to them to make their own games. The second 
strategy involved designing a custom game from beginning to end, with students building their own overall 
structure, objects, characters, and rooms. Among the 8 games examined, two students used the templates to build 
their games and the remaining 6 designed their own games without using templates. The latter strategy is 
considerably more complex. 

Overall, all students kept with their game theme over time. That is, once students started work on their 
games, they did not alter the overall theme and design. Even though students made changes to the games by adding 
new features throughout the project, the overall concept and theme of the games remained. This finding differed 
from prior research conducted by Kafai (1996) that showed that students made changes in overall game theme and 
concept. 

Nevertheless, the students’ games reflected substantial progress in terms of students’ design skills over a 
short time. For example. Flora indicated that she played games often but had never before used game design 
software. Yet, by the end of the project, she was able to design an educational game that reflected nutrition concepts. 
The characters in her game not only performed basic moves but also functioned to make the player lose points when 
the main character touched “unhealthy food”. She also added interactive feedback, a more advanced function that 
few students included, to guide the player of her game. 

Analyses of interviews, artifacts, and observations showed that the students used various strategies to 
design their games. One of the strategies that all the students used was exploring the coding examples in the 
template games. Since the students could to access the coding part of the template games, they were able to see how 
each coding feature made the game different. The students used that resource wisely. However, that strategy may 
have also led the students to pursue game themes and ideas that were driven by those available as templates. 

Another strategy that aided in developing programming skills was accessing a teacher. The teachers acted 
as facilitators in the design process. Instead of teaching every step of how to design a game, the teacher showed the 
students how to access necessary information. The technology teacher was a useful resource when students were 
seeking technical help. Moreover, the teacher promoted peer collaboration by referring students’ questions to other 
students who might have good answers for a specific problem. This type of peer collaboration was the dominant 
strategy used to design and build upon games. Learning from peers helped students not only learn basic coding from 
each other but also to collaborate on game designs throughout the project. For example, Alan was the only student at 
the start of the project who knew how to write code in Gamemaker™ . However, with Alan’s help, most of the 
students added some coding to their game designs. Collaboration in game design seemed to influence students to 
add new features to their games. 

Collaboration among the students was not limited to the game design process. The students also asked 
questions to each other regarding nutrition. For instance, we observed some students asking others if they had proper 
portion sizes of certain foods in their games. Another way of collaboration during the design project was students’ 
testing each others’ games. For example, one student (Erin) said: “During class, they were like, ‘Erin’! ‘You wanna 
come and see? Do you like this?’ and things like that..." Testing each others’ games helped students provide peer 
feedback and to build new features into their games, as a result of being exposed to new ideas. In addition, testing 
games was useful to see the errors in coding, giving them the chance to make necessary changes to their games. 
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Gender Differences 

Gender differences were analyzed according to the following game characteristics identified by Kafai 
(1996) and Robertson (2004); (a) game genre (b) game world and environment (c) game characters and (d) 
interaction and feedback. 

Game genre. The genre all of the students’ games can be categorized as arcade. All of the games were 
comprised of several levels where each level was progressively harder for the players. The average number of levels 
in the games was 5.2 for the girls and 6.7 for the boys. 

Similar to other commercial games, such as Pacman, players’ goal in all of the students’ games was to 
keep the main characters alive to finish the game. The game would end if the game character loses life or a certain 
number of points. In this study, gender differences appeared with regard to the display of scoring for the players. All 
the girls used either scoring or live counting for the players, but only two boys used one of them. 

Gender differences also appeared in the methods used to save characters. All the boys’ games involved 
destroying enemies or unhealthy foods to make the main character survive. For example, Tom’s character shoots the 
brownies. Markus’ stays away from enemies and makes them crash. Sammy’s character also destroys all the “bad 
foods” by touching the magic food. On the other hand, none of the girl games involved destroying characters. In 
order to survive, the players of the girls’ games had to only stay away from the “unhealthy food” and get the 
“healthy food”. 




Game world. The students’ game worlds, or as Robertson (2004) describes as “game settings”, had several 
commonalities. Interestingly, in this study none of the students used a real world environment as the setting for their 
games. Most the games took place in an imaginary setting with boundaries around which the character bounced. 
Otherwise, almost all the environments allowed the players to move in four directions (left, right, up, and down). 
The one exception was Markus’ game, and his game was a food driving game that was modified from a car driving 
template game. The environment in his game was similar to a highway where food icons tried to escape from the 
“bad food” and drive carefully. 

Development of game character. In the games, all the students chose a main character that represented 
the player whose goal was to eat “healthy foods” and stay away from “unhealthy foods”. Gender differences 
appeared with the students’ characters in their games. The characters that all the girls, except Amanda, used were 
not ones that were used in commercial games. Alternatively, the characters that boys used, except Tom’s characters, 
were game characters from commercial games. Table 1 summarizes characteristics of the students’ game characters. 
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Table 1: Students game settings and characters 



Student 


Main 

Character 


Supporting Characters 


Design/ 

Gallery 


Character 

Gender 


Known character 


Carol 


a stick man 


8 characters and 1 of them 
moves 


Her own 
design 


No gender 
specified 


Not for the games that 
she had played 


Flora 


Initial of her 
name in a 
circle 


20 characters and 2 of 
them move 


Her own 
design 


No gender 
specified 


Not for the games that 
she had played 


Erin 


Teddy bear 


3 1 characters and 1 of 
them moves 


From the 

image 

gallery 


No gender 
specified 


Not for the games that 
she had played 


Amanda 


Pacman 


Added 7 new characters 
to the original game; 2 of 
them move 


From the 

image 

gallery 


No gender 
specified 


Common character 


Tom 


Hamburger 


12 characters and 5 of 
them move 


From the 

image 

gallery 


No gender 
specified 


Not for the games that 
he had played 


Sammy 


a Pokemon 
character 


13 characters and 3 of 
them move 


From the 

image 

gallery 


No gender 
specified 


Common character 


Markus 


Pacman 


Added 3 new characters 
to the original game; 4 of 
them move 


From the 

image 

gallery 


No gender 
specified 


Common character 


John 


A bird 


14 characters and 2 of 
them move 


From the 

image 

gallery 


No gender 
specified 


Common character 



Based on our analyses, some gender difference emerged in structuring the characters. Girls had an average 18.2 
characters and in average 2.5 of them moved, (total of 6) yet the boys had an average 12 characters and average 14 
and in average 3.5 of them moved (total of 14). Having more moving characters made the boys games not only more 
challenging but also more interactive. However, we also found that none of students designed a main character with 
a specified gender. 

Interaction and feedback in the games. In most of the students’ games, the feedback for the players was 
based on action rather than text feedback. When players lost the game, most of the students required the players to 
either restart the game from the beginning or end the game by showing the score. Except for two girls. Flora and 
Erin, no students gave any direction for the players to explain the steps in the game. However, these two girls tried 
to inform the player and provide brief feedback when the games ended. For example, Erin’s game started with the 
following message: ‘'Avoid the blue ghosts and collect all of the balls”. When the game ended, the message stated; 
“Sorry you lost. Come back and try again later!” 

On the other hand. Flora added a character that resembled Einstein to guide the player throughout the 
game. In her game, when the main character touched that character, the following message popped-up; “Hi my name 
is Bob... When you’re done, come to me!” Flora’s games ended with the standard “Game Over!” message. 

Conclusion 

This study investigated how children designed computer games as artifacts that reflected their 
understanding of nutrition. Our study showed that the students were able to successfully express their personal 
thoughts and intentions by designing and developing computer games in a complex programming environment. 
Although the focus of our analyses was on the children who designed the games, the first graders who played them 
also reportedly enjoyed the games. They expressed awareness of which foods were healthy and unhealthy in the 
game, and indicated that they wanted to play the games again during the try-out period. 

One noteworthy aspect of this study relates to the girls’ engagement in designing computer games. The 
girls in our project designed computer games, as opposed to just playing games. Also important is the fact that some 
of the girls continued to design computer games in after-school activities after the project ended. Erin, for instance, 
continued designing computer games six months after the project ended, even though she stated that she didn’t play 
computer games. In the end, the girls’ games were as educational as boys’ games. Our study findings support the 
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idea that boys are likely to see computers as a playful recreational toy, whereas girls see them as a tool to 
accomplish a task (Miller, Chaika, & Groppe 1996). 

Nevertheless, the project also dealt with some challenges. Since it was largely the students’ first time 
designing computer games with Gamemaker™, the students had technical difficulties to implement the ideas they 
planned. Some students, for instance, had to draw their own food icons since the icons they wanted to use in their 
games either were not provided or were oversized for their games. Like other studies of students developing 
artifacts, the process of making the artifacts themselves sometimes becomes the overwhelming focus, rather than the 
knowledge goals of the activity (Barron et ah, 1998). Future studies could focus on addressing issues of 
measurement - particularly in regard to assessing the knowledge gains associated with such an activity. 
Notwithstanding, our findings provide some insights into the process used by students to design computer games 
that reflect nutrition concepts. Our preliminary data support the notion that learning by designing computer game 
artifacts can be beneficial for students’ learning, regardless of gender, and also can lead to productive social 
interaction among the students. 
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Abstract: The University of Hawaii launched a statewide initiative to incorporate technology into teacher education. 
This qualitative study examines faculty members’ progress in learning to integrate new technologies in relation to 
technology content standards. Seventy-eight faculty members provided course artifacts and written narratives before 
and after their participation in a technology initiative utilizing technology mentoring. Findings suggest only faculty 
mentored by individuals with degrees in Instructional Design discussed their technology use within the framework 
of standards. 



Background and Introduction 

The United States federal government passed legislation in 1994 establishing national standards for the use 
of technology by students ages five through eighteen and their teachers (US Congress OTA, 1995). This legislation 
presented an enormous challenge to both public schools and colleges of education to incorporate technology skills 
and pedagogy when preparing current and future teachers. 

The Preparing Tomorrow’s Teachers to Use Technology (PT3) program was established in 1999 as a 
subsidiary program of the United States Department of Education (US DOE). Through this one program, the US 
DOE invested a total of nearly $400 million into the capacity-building of teacher education programs to instruct 
preservice teachers on how to use technology effectively to enhance student learning. This technology integration 
aim was approached through over 400 programs between 1999-2003; each selecting their own project focus, 
activities, and plans. A common theme among many projects was the goal of offering professional development to 
higher education faculty teaching in preservice teacher education programs (PT3, 2002). 

Professional development programs are tasked with trying to speed individuals through various stages of 
learning, as quickly as possible, while preserving the quality of instruction and the long-term adoption of a 
technology. Numerous professional development programs exist to help faculty members learn technology. 

Different strategies have been proposed and utilized in professional development. However, despite the proliferation 
of technology-introducing strategies offered for faculty at universities throughout the world, there is no clarity in 
how these programs compare, which is better suited to a particular type of institution, or which produces stronger 
results. 

While a decisive answer is not currently within grasp, mentoring has been proposed to be a promising 
strategy. Chuang, Thompson, and Schmidt (2003) summarize major trends in the literature on faculty technology 
mentoring programs, describing programs using graduate students as mentors, undergraduate students as mentors, 
and K-12 pupils as technology support for inservice teachers. No peer colleague mentoring programs were 
described. A significant contribution of their review is their common themes of strong mentoring programs. 

“Despite the variety of technology mentoring models, effective programs include common elements . . . 
These elements include providing visions for the use of technology in teaching and learning, 
individualizing technology support (personal fit), breaking down hierarchical structure, establishing open 
dialogue and collaborative relationships, providing mutual benefits for mentors and mentees, and 
establishing learning communities” (Chuang et ah, 2003). 

Building on this growing understanding of technology mentoring programs, recent research extends the 
field’s understanding of the complexities related to technology mentoring (Sprague & Cooper, 2004). At George 
Mason University, faculty members were paired with K-12 teachers noted for their technology use. Widmayer 
(2004) details three pairs over 12 to 18 months. Widmayer’ s case study approach provides insights into what these 
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faculty members were working on and the evolution of their progress; however, there may be some question as to 
whether the mentoring in this study meets the recommended criteria put forth by Chuang et al. (2003) for strong 
mentoring programs. Research conducted at Iowa University is providing the most direction in current investigation 
of faculty technology mentoring. Iowa’s early research was critiqued because it relies on case study protocols, that 
is, student products created as part of their university course in which they mentored a faculty member as a course 
assignment. However, Chuang (2004) explores a faculty member she mentored over an eight-year period through a 
reflective case study approach. These initial attempts should be commended for their ground-breaking work and 
their honesty in chronicling what does happen when faculty are mentored to use technology. These attempts further 
encourage the field to delve into this relatively under-explored area. To date, these accounts predominantly rely on 
self-reported or mentor-reported accounts of technology integration. Further, the focus is often on faculty’s attitudes 
in relation to technology. Data is typically in the form of surveys or interviews with faculty or mentees. In other 
words, faculty members or the mentees reflect on what the faculty perceive they have gained from mentoring, for 
example, more confidence. 

A study is needed that extends this line of research to include evidenced-based data; what can one see is 
actually changing in courses and teaching practices while participants are in a mentoring program? Studies are 
needed that target the faculty member’s technology skills versus only their attitudes, while recognizing that these are 
to some degree related. While research has come a long way in the last couple of decades, the understanding of the 
outcomes of technology mentoring programs is still in a fledgling stage. With dwindling public funds for higher 
education, administrators are forced to examine professional development programs for faculty with more scrutiny. 
However, little is understood about the qualitative differences in faculty’s courses and their own skill sets after 
professional development. The cost effectiveness of such programs is important to consider, but cannot be 
considered in a vacuum without developing a better understanding of the actual results of such programs. 

This qualitative study examines a particular outcome of a professional development initiative. The 
University of Hawaii, through two Preparing Tomorrow’s Teachers to Use Technology grants, launched a statewide 
initiative to infuse technology into teacher preparation. A major component was to share practices and build inter- 
institutional collaborations. A partnership was established between the College of Education and each of the seven 
state community colleges, as more than half of the students entering the College of Education have coursework from 
these institutions. Over a two-year period, 78 faculty members from all the community college campuses 
participated in this technology initiative that included technology mentoring. This study examines the faculty’s 
beginning attempts at integrating technology and how those relate to technology content standards. The technology 
standards utilized in this study are known as the Technology Intensive standards, developed for use in the University 
of Hawaii system. Eaculty from the Educational Technology Department served on committees involved with the 
creation of national technology standards that went on to become modified and adopted by the International Society 
of Technology Education (ISTE technology standards). Therefore, the technology standards adopted locally relate 
more closely to the national technology standards than other standards created on a local level (Eulford, 1997). 

Conceptual Eramework of Technology Mentoring 

The Department of Educational Technology (ETEC) led the University of Hawaii at Manoa’s College of 
Education (COE) in responding to the national concerns related to integrating technology into preservice teacher 
education. Eulford and Ho (2002) developed a model of institutional change for technology integration based on 
faculty mentoring. The Technology Intensive (TI) Courses Model used “technology mentors’’ to assist COE faculty 
with technology integration (Eulford & Eichelberger, 2003). The approach was based on the philosophy that 
teachers will teach the way they are taught. Therefore, in order to teach preservice teachers to effectively use 
technology, COE faculty needed to model effective technology use in their own teaching and require their students 
to use technology in their academic work and research. The goal was to create teacher education faculty who were 
technology role models and technology-savvy new teachers. The TI model used a systematic approach for faculty 
recruitment, course redesign, and evaluation to assess the integration technology into courses. 

The TI Model consists of two phases. The first phase of the model uses a number of strategies to recruit 
faculty to the technology mentoring program. The second phase, involves technology mentors assisting faculty one- 
on-one in the redesign of their courses to integrate technology. To meet the challenge of reluctant faculty, Eulford 
and Ho (2002) developed a three-tiered approach to redesigning courses to incorporate technology. The first and 
most introductory level is Technology Enhanced Courses. In these courses, the faculty member demonstrates and 
uses technology in the presentation of course content and communicates with students electronically. This level 
strives to be the least threatening to faculty who are just starting to use technology in the classroom. The second 
level is Technology Applied Courses. These courses add the use of technology to the current course structure 
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encouraging students to use technology resources in their research, communication, and presentations. The faculty 
member demonstrates and uses technology in the presentation of course content and communicates with students 
electronically. The third level is Technology Intensive Courses. These courses follow the Technology Intensive 
Standards and Guidelines to improve technology literacy while continuing to emphasize course content (Fulford & 
Hines, 1997). Students have a high level of involvement using technology, while faculty meet standards as 
exemplary role models using technology. 

The three levels allowed faculty to become involved with the project and move at their own pace with 
technology integration. The levels also allowed for faculty to decide the appropriate level of technology integration 
for specific courses, as some courses may not be appropriate for the Technology Intensive designation. Allowing 
for flexibility and a range of technology usages was perceived to be key in making the Technology Intensive 
Courses model work for many faculty members in a variety of subject areas. 

In order to redesign their courses to any of the TI levels, faculty needed assistance revising course 
objectives, creating new strategies and activities, rethinking student projects, locating and creating new media, and 
developing alternative assessments. Faculty who expressed interest in integrating technology into their courses were 
offered one-on-one mentoring assistance to facilitate this process. Participating faculty were paired with a 
technology mentor. Typically, the pairs met on a weekly basis over a semester. Many faculty continued mentoring 
past one semester, often with the same, but sometimes different, mentor. The mentoring sessions usually lasted 
about an hour and took place in the faculty member’s office. 

In order for the mentor to be able to effectively guide the faculty toward using technology in their teaching, 
an instructional design approach was used to: (a) set goals for the professional development; (b) provide expertise in 
creating a revised curriculum especially with regard to the technology-intensive standards; and (c) assist in 
improving technical skills to help faculty members reach their technology integration goals. In the mentoring 
process, the mentor first assessed the technology needs and levels of confidence of their mentee via a technology 
skills survey. The pairs then discussed the course content and goals the faculty member had for redesigning the 
course to integrate technology. The mentor had to be tactful with the faculty member to be encouraging and yet 
realistic as to what could be accomplished in one or two semesters. The pairs discussed their respective time 
schedules and the amount of time the faculty member had to dedicate to the redesign process. A contract was signed 
between the mentor and mentee to encourage dialogue about the role of the mentor and mentee. 

Technology mentors needed to be thoroughly trained in order to provide both the technology skills and 
instructional design knowledge to help the faculty member plan and organize the course to be redesigned. The 
mentors were trained to assist faculty members to become independent, self-sufficient users of technology. To 
achieve this goal, students often needed to know how to handle difficult situations. Examples of difficult situations 
included faculty members expecting the mentor to do the technology work for them, faculty members not 
completing work they have agreed to do in order for the sessions to proceed, chronic appointment cancellations, and 
faculty who were continually distracted by new media and strayed off topic every week. 

In order to conduct one-on-one sessions and handle these situations, new technology mentors participated 
in a series of training sessions led by a supervising faculty member and graduate students who had mentoring 
experience. Training topics included the Technology Intensive Standards and Guidelines (Fulford & Hines, 1997), 
customer service, specific hardware and software, and perhaps most importantly, “how to conduct one-on-ones,” a 
training session in which trainers role-played and discussed solutions for a variety of difficult situations. Sessions 
further introduced new mentors to the materials created to assist the mentoring and course redesign process. These 
“cross-training” sessions were offered as part of an on-going monthly work group. 

The background and summary of the TI model (Fulford & Ho, 2002) is provided as a professional 
development framework influential in Hawaii. Further, this detailed program description is important to compare to 
literatures on effective mentoring programs and relationships. Also, it should assist readers in understanding the 
systematic nature of formal mentoring programs developed to encourage the use of technology. 

Methodology 

This qualitative study explored 78 faculty members who participated in a statewide initiative to incorporate 
technology into community college courses. Participants taught at seven different institutions of higher education in 
the state of Hawaii. Faculty participants who volunteered entered a mentoring relationship with a technology 
mentor from their same campus. The goal of these mentoring pairs was to improve the technology skills of the 
mentee and enable each mentee to begin to integrate technology into their courses. Across all campuses, mentoring 
was used as a professional development strategy to increase the technology skills and integration of the mentees. For 
a more detailed description of the mentoring relationships, mentoring program structure, learning community of the 
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mentors, or institutional contexts refer to Boulay (2008). In general, participants were paired with a technology 
mentor who worked with them for one or two semesters to learn new technologies and integrate those technologies 
into their curricula. This study triangulated data from multiple methods and sources: 1) course products and 
instructional materials, 2) written narratives describing faculty’s familiarity and use of technology in courses, and 3) 
site visits to institutions and discussions with participants. 

The 78 faculty members in this study submitted narratives and course products before and after 
participation in the technology initiative. Participants provided course artifacts that demonstrated their initial 
technology skill level. As mentoring progressed, faculty submitted new or revised course artifacts that demonstrated 
their progress on a given project. Course artifacts often included screen shots of online course pages or websites, 
newly created digital presentations, syllabi or handouts, student assignments, scanned work, or PDF versions of 
documents. The most significant and exciting documentation comes from courses in which faculty require students 
to use technology and included examples of student work. These course artifacts paint the picture of what the 
faculty member was working on, what they were creating, and how they were revising their course with concrete 
examples; the artifacts were intended to demonstrate the impact of mentoring on faculty’s teaching materials. The 
artifacts also proved useful in recruiting new faculty by demonstrating the possibilities of the Technology Intensive 
mentoring program; the course artifacts were used to corroborate the data of the narrative descriptions. In addition 
to course artifacts, written narrative descriptions of faculty members’ technology skills and the use of technology in 
their courses were collected. Narrative descriptions ranged from two to six pages and were collected before and after 
mentoring. 

Data analysis for this qualitative research study was approached through multiple strategies. A researcher 
examined and reexamined all of the narrative descriptions and course artifacts to identify patterns and themes. The 
qualitative comparative method of data analysis (Ragin, 1987) was used to construct categories and themes that 
captured the recurring patterns that emerged from the data. The analysis of the data was cyclical, consisting of 
initial coding, reflecting, and re-reading, then sorting and sifting through the codes to discover patterns and themes. 
These methods were used to triangulate the evidence of the data (Lincoln & Guba, 1985). Since the intent of this 
paper is to describe how faculty progressed with regards to using technology in relation to technology content 
standards, the multiple data sources were examined for relevant comments or codes relating to technology content 
standards. This particular study used methods triangulation and triangulation of sources (Patton, 1999; Patton, 

1990). By combining multiple methods and data sources, this study provides a robust design in comparison to those 
using singular methods. The principal researcher aimed for an analytically rigorous, mentally replicable, and 
explicitly systematic approach to data analysis (Patton, 1999, p. 1191). 

Research Questions 

The course products and narratives were analyzed after being collected to investigate 1) how many faculty 
participants discussed their technology use in relation to technology content standards, 2) which technology content 
standards were addressed, and 3) what patterns were salient in the use of technology aligned to technology content 
standards among participants? 



Results 

Of the 78 faculty in this study, approximately a quarter of the sample population self-identified the 
technology standards they were attempting to address in their courses. Eighteen faculty participants named the 
standards and discussed the activities they and their students were doing to address each standard. Despite all 
technology mentors being instructed on the technology standards as a guide, only faculty located at campuses with 
high-level technology mentors, who had advanced degrees in Educational Technology from programs that 
emphasize instructional design, identified the technology standards they were addressing in their courses. 

The faculty identified 28 of the total 33 technology content standards within their teaching. See Eigure for 
a list of the technology standards and the number of faculty who stated that they addressed this standard within their 
course. Additionally, eight faculty participants included at least one technology standard from each of the six 
categories. This suggests that among faculty who identify their course as addressing technology standards, 
approximately half of them are covering the recommended breadth and including standards from the multiple 
categories. 

From the sample of faculty who identified their course design as addressing technology content standards, 
faculty indicated that their course activities most often related to the following standards: Operations 2.2, Retrieval 
4.1, and Application 5.2. These three standards: operating a multimedia computer and peripherals, using automated 
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on-line search tools, and utilizing telecommunications tools such as electronic mail and web browser applications 
for communications and research, were present 70% of the time among the faculty who identified technology 
standards. The five standards that were not addressed by any of the faculty were spread among four distinct 
categories of standards, that is, 1.5 Ethics, 2.6 Operations, 5.7 and 5.8 Application, and 6.8 Attitudes. While only 
five standards were omitted, more than half of the standards were addressed only 10% of the time or less by those 
faculty indicating that their course addressed technology standards. This means only 14 of the 33 standards were 
identified by more than two faculty, who identified technology standards their course addressed. Two or fewer 
faculty participants identified the other 19 of the 33 standards. 

While a substantial subsection of faculty identified standards in their course syllabi or course proposals, 
only one faculty mentioned standards in her reactions or narrative comments. 

“I had never seen standards and didn’t know they existed. They really helped me think and evaluate about 
what I was requiring students to do. They provided at least an outline of what might be good for students 
to do with technology and then I could evaluate what components I could accommodate in my course” 
(notes from site visit, participant 051). 



#of 

Faculty 


Technology Intensive Standards 




1.0 Ethics: Use print and electronic technology ethically and responsibly. 


1 


1 . 1 Analyze and describe the social implications of advanced technology and the ways 
representations of human-technology relationships shape social attitudes. 


2 


1 .2 Assess the challenges posed by technology. 


1 


1 .3 Describe technology as it shapes society and the environment. 


1 


1 .4 Describe the cross cultural implications of the use of technology and generate personal 
standards of ethical use. 


0 


1 .5 Describe strategies for facilitating consideration of ethical, legal, and human issues 
involving purchasing and policy decisions, (adapted from ISTE-2.1.2) 


5 


1 .6 Seek information about current copyright and patent laws and abide by ethical standards 
of the use and transfer of information. 


4 


1.7 Use technology to improve communication that addresses the diversity inherent in people 




2 . 0 Operations: Use basic vocabulary and concepts, and operate technology. 


3 


2.1 Use terminology related to computers and other electronic technology appropriately in 
written and oral communications. (Adapted from ISTE-1.1.2) 


13 


2.2 Operate a multimedia computer with related peripheral devices. (Adapted from ISTE- 
1.1.1) 


4 


2.3 Use imaging devices such as scanners, digital cameras, and/or video cameras with 
computer systems and software. (ISTE-1.1.4) 


2 


2.4 Install application software and peripheral devices and their accompanying software. 


4 


2.5 Use a variety of technologies such as video cameras, fax machines, and copy machines 
to enhance communications. 


0 


2.6 Describe and implement basic troubleshooting strategies when using equipment such as, 
multimedia computers, peripheral devices, video cameras, fax machines, and copy 
machines, (from ISTE-1.1.3) 




3.0 Analysis: Recognize, identify, and define an information need. 
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7 


3.1 Describe different modes of inquiry and information acquisition. 


4 


3.2 Discuss advantages and disadvantages offered by differing forms of technology in 
solving information issues. 




4.0 Retrieval: Access and retrieve information through print and electronic media, 
evaluating the accuracy and authenticity of that information. 


15 


4.1 Use automated on-line search tools and intelligent agents to identify and index desired 
information resources. (ISTE-2.3.3) 


5 


4.2 Check references and evaluate information for validity and reliability. 




5.0 Application: Create, manage, organize and communicate information through 
electronic media. 


1 


5.1 Apply relevant technologies to one’s professional field. 


13 


5.2 Use telecommunications tools such as electronic mail and web browser applications for 
communications and research. 


2 


5.3 Create and display audio-visual presentations and/or multimedia/hypermedia productions 
that are equal in sophistication in form or content to a well written, paragraph, essay, 
monograph or novel. 


4 


5.4 Use and interpret visual information and describe how it affects meaning. 


2 


5.5 Use visual design techniques to maximize readability, legibility, and accuracy in 
information presentation and display. 


2 


5.6 Use technology for problem solving. 


0 


5.7 Process, analyze, interpret, and communicate information using electronic analysis 
software such as spreadsheets, statistical packages, management, and databases. 


0 


5.8 Envision, shape and create new technologies. 




6.0 Attitudes: Recognize changing technologies and make informed choices about 
their appropriateness. 


8 


6.1 Choose to explore and use a variety of information technologies to enhance their 
personal and professional lives. 


2 


6.2 Utilize technology tools to address multiple intelligences and representations. 


0 


6.3 Make informed choices about purchases and use of technology. 


1 


6.4 Choose to be a self sufficient technology user by accessing help menus, manuals, and on- 
line documentation rather than have a primary reliance on outside personal assistance. 


5 


6.5 Create, manage, and organize ideas and feelings through various media. 


1 


6.6 View technology as requiring continuous education to keep current. 


1 


6.7 Seek and use information regarding the impact of technology on health and well being 
with regard to physiological, psychological and social issues. 


2 


6.8 Use electronic management tools to organize and balance the use of personal and 
professional time in a way that will maximize health and well-being. 



Figure. The Technology Intensive Standards identified by faculty participants. 
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A quarter of faculty participants in this study were beginning to discuss the technology that they and their 
students used within a context of technology content standards after their participation in a technology initiative that 
utilized a mentoring component. Their and their students’ technology usage was emerging to begin to meet 
technology standards. Faculty in the sample over-emphasized common teaching practices or instructional activities 
to address standards and too heavily focused on a finite set of three standards. Also, the depth of mastery of 
standards is questionable. However, these faculty members only participated in these technology initiatives for a 
short period of time. Their redesign of curriculum to address technology content standards may be in beginning 
stages and continue to emerge. An unanswered question is whether personalized mentoring can increase the number 
of faculty who intensively focus on technology in that they tailor specifically how they and their students use 
technology in order to meet technology content standards? As a quarter of faculty participants in this sample began 
to experiment, conceptualize, and incorporate technology in relation to technology content standards, mentoring 
seems to provide at least one possible avenue for increasing faculty’s awareness and experimentation of standards- 
focused inclusion of technology into their teaching and student learning outcomes. 

Discussion 

Ultimately, the national community of professionals desires faculty who meet the national educational 
technology standards for teachers and students, as these standards represent a collective agreement about how 
individuals should be using technology to meet teaching objectives (ISTE, 2000, 2002, & 2005). This study found 
that technology mentors, with an advanced degree in Educational Technology from a program with a strong 
instructional design emphasis, in addition to, having advanced technology skills, shaped faculty to conceptualize and 
articulate their technology use in relation to technology content standards. Approximately, a quarter of the sample 
population self-identified the technology standards they were attempting to address in their courses. These faculty 
participants selected the standards and discussed the activities they and their students were using to address each 
standard. The Technology Intensive Model (Eulford & Ho, 2002) suggested that if faculty were allowed the 
autonomy to select which standards to address, the majority of standards would be addressed in a student’s 
education pathway. The data generally supported this claim. The faculty identified 28 out of the 33 standards and 
standards in each of the six categories: ethics, operations, analysis, application, and attitudes. 

Eaculty overemphasize specific standards 

Hall (2006) suggested that faculty address certain categories or standards more extensively than other 
categories. Hall (2006) identified Ethics as a particular category of standards under-emphasized among participants 
in her institutional case study. This study drew similar but not identical conclusions. Eaculty in this study were just 
as likely to include Ethics, and often emphasized this category. Among the faculty who identified technology 
standards, 70% identified the same three standards: operating a multimedia computer and peripherals (operations 
2.2), using automated on-line search tools (retrieval 4.1), and utilizing telecommunications tools such as electronic 
mail and web browser applications for communications and research (application 5.2). This study suggests that 
rather than certain categories being ignored, faculty may overuse the same activities to achieve the same limited 
number of standards. In this study, participants were approaching students addressing these three standards using 
predominantly the same student activities versus using a wide variety of assignments or activities challenging 
students to address standards in multiple ways. Perhaps faculty over-utilize certain standards because they relate to 
technologies the faculty have heard of and want to try. 

Eaculty omit some technology standards 

In general education courses, faculty did not emphasize technology standards related to policy, purchasing, 
and evaluating resources. The five standards not identified within the sample are the following: 

1 . (Ethics category) Describe strategies for facilitating consideration of ethical, legal, and human 
issues involving purchasing and policy decisions, (adapted from ISTE-2. 1 .2) 

2. (Operations category) Describe and implement basic troubleshooting strategies when using 
equipment such as, multimedia computers, peripheral devices, video cameras, fax machines, and 
copy machines, (from ISTE- 1.1. 3) 

3. (Application category) Process, analyze, interpret, and communicate information using electronic 
analysis software such as spreadsheets, statistical packages, management, and databases. 

4. (Application category) Envision, shape and create new technologies. 
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5. (Attitudes category) Make informed choices about purchases and use of technology. 

These standards align less clearly with content areas, such as English, Math, or Science, perhaps than the other 
standards. Where these standards should be addressed in a student’s educational pathway may cause debate. These 
standards are included in courses designed to teach students about technology; however, as technology is more 
infused into regular coursework and offered less as an individual subject, these standards may not be integrated into 
those courses without specific attention to incorporate them. 

Standards broaden teaching approach 

Technology mentoring seemed to provide at least one possible avenue for increasing faculty’s awareness 
and experimentation of standards-focused inclusion of technology into their teaching and student learning outcomes. 
Faculty, initially, had a confined set ideas of how to use technology; using technology content standards in 
professional development led to faculty expanding that original set of ideas. A quarter of faculty participants in this 
sample began to experiment, conceptualize, and incorporate technology in relation to technology content standards. 
By introducing technologies into a course for the first time, faculty naturally seemed to engage in dialogue and 
critical thinking about how and why to use particular activities and instructional techniques. As they thought about 
how to replicate student activities or assignments using a new medium or capitalizing on a newer technology, they 
made adjustments to their course or teaching based on that reflective process. Examples among the dataset can be 
highlighted to demonstrate technology revolutionizing how certain individuals taught. Common changes were 
observed, for example, using technology to increase student-teacher interaction, making samples and previous 
student examples available to students, increasing efficiency or course management. 

This study found that some faculty stated providing technology standards increased their awareness of 
desired technology uses in student learning. By using technology content standards, within a professional 
development approach, faculty learned about new categories of technology use, they were not overtly aware of, prior 
to participating in a technology initiative. The standards stretched the thinking about technology, among faculty 
participants, to include broader student objectives for technology usage. A technology standards-based framework 
assisted participants in conceptualizing curricular revisions to address a greater number of technology content 
standards. 

Call for evolution of standards discussion 

A recent discussion in the literature has ensued about assessment, and an important dichotomy has been 
suggested between the “assessment of learning’’ and “assessment fw learning’’ (Klenowski, 2000; McMullan et ak, 
2003; Smith & Tillema, 2003; Stiggins, 2002). Assessment of learning documents what has been learned and tends 
to relate more often to summative assessments. Assessment fw learning is used for the purpose of educating the 
learner and their teacher about where the learner is presently and helps to identify target areas for growth. 
Assessment fw learning tends to be used more in formative assessment circumstances (Stiggins, 2002). The 
standards that participants did not address or findings about overemphasizing a certain subset should be taken as part 
of an assessment fw learning. 

Lessons gained from this study could inform participants and others about faculty’s initial attempts to use 
technology in courses and how those attempts relate to technology content standards. From the learner’s 
perspective, they are displaying their accomplishments. The faculty participants in this study went from not 
identifying technology standards to eighteen identifying technology standards that students met as part of 
completing the activities and assignments in their courses. This accomplishment is remarkable, given that only four 
faculty were integrating technology in any way that encouraged student use of technology prior to participation in 
this technology initiative that used technology mentoring. 

While studies, such as this one and Hall (2006), initiate dialogue about technology integration and how 
faculty are using technology and requiring their students to use technology, these studies report what standards 
faculty state they are addressing. A more careful analysis into how faculty and courses are organized to assist 
students in mastering specific standards is necessary. The discussion needs to move beyond whether faculty are or 
are not addressing a standard. As a field, there is a need to engage in a more provocative discussion about unique 
ways to enable all students to meet a particular standard or more beneficial approaches to enable a deeper mastery of 
standards by students. 
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Conclusion 



This study examined the outcomes of a two-year professional development program initiative that utilized 
mentoring to encourage faculty to incorporate technology into curricula. The revised course products and narrative 
descriptions of 100% of the 78 faculty participants were examined to critically analyze how the professional 
development activities had influenced incorporation of technology aligned to technology content standards. A 
quarter of faculty participants in this study were beginning to discuss the technology that they and their students 
used within a context of technology content standards after their participation in a technology initiative that utilized 
a mentoring component. Their and their students’ technology usage was beginning to meet technology standards. 

An interesting pattern emerged that only participants who had worked with a technology mentor who had 
an advanced degree in Educational Technology from a department with a strong instructional design emphasis chose 
to describe their technology use in connection to technology content standards. This may suggest that strong 
technology skills among technology professional developers are not sufficient to promote standards-aligned 
incorporation of technology into courses. 

In critical reflection, faculty in the sample over-emphasized common teaching practices or instructional 
activities to address standards and perhaps too heavily focused on a finite set of three standards. Also, the depth of 
mastery of standards was not examined. However, these faculty only participated in this technology initiative for a 
short period of time. Their redesign of curriculum to address technology content standards may be in beginning 
stages and continue to emerge. Examinations of the outcomes of professional development are critical in the 
development and sustenance of successful practices, especially in current financially restrictive environments in 
higher education. 
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Enhancing learner motivation through goal messaging and goal orientation 



John M. Bunch 
Abstract 

The current study investigates instructional design factors that can be manipulated to enhance learner motivation. 
Current theory concerning the goal orientation an individual learner brings to an instructional situation is predicted to 
impact any attempt to influence learner motivation through instructional design. In addition, the efficacy of Prospect Theory 
as a cognitive mechanism underlying the valuation of effort toward a goal is discussed, and an experiment is presented in 
which goal messaging is manipulated based on the predictions of Prospect Theory as well as Goal Setting Theory. A Web- 
based tutorial consisting of ten sections of text, each with a recall test, was used. An ability goal orientation was found to 
impact section quiz scores. Both goal message framing and goal difficulty level were found to interact with an ability goal 
orientation to impact performance on section quizzes. A learning goal orientation was found to interact with goal difficulty 
to impact section quiz scores. The author concludes that while the study supports the use of goal messaging to enhance 
motivation, such manipulations by educators must be made in light of the goal orientations a learner brings to the 
instructional setting. 

Introduction 

This study investigates the use of performance goal messages to enhance learner motivation in a Web-based 
leaning task. Two theoretical approaches, neither of which have been applied in an instructional context within the 
educational psychology/technology literature, are considered with regard to the predictive guidance they may provide for 
this endeavor. 

First is Goal Setting Theory (GST) (Locke & Latham, 1990), perhaps the most influential model of motivation to 
have emerged in the literature outside of education. GST emerged from hundreds of laboratory and field studies, primarily 
related to work and work performance in corporate settings. GST holds that performance can be enhanced by manipulating 
the goals set for workers, and that there is a positive, linear relationship between goal difficulty and performance, even in 
situations where the goal is not reached. 

The second theory is Prospect Theory (Kahneman & Tversky, 1979), originally developed as a model of risky 
decision making in economic situations, and since applied to a wide variety of behavioral phenomenon including consumer 
behavior, politics, and health care. Heath, Larrick, & Wu (1999) provide evidence that a human performance goal acts as a 
reference point in Prospect Theory's value function, and thus Prospect Theory can be used to predict the value of effort 
toward a goal, depending on how far an individual is from the goal, and whether the individual is above or below the goal. 

Here's how it works. Consider Figure 1 below; 

Figure 1. Differences in value function slope. 
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The value function has an S shape - so it's steeper in some parts than others, and has a different shape below the 
goal than above it. So, if student Joe has a goal of scoring 135 cumulative points on 10 quizzes, the value of effort at Quiz 9 
is greater than the value of effort at Quiz 3. If we were to frame the grade scenario differently, however, and speak in terms 
of Joe trying to avoid losing points on each quiz from an original point total higher than the goal, notice that the value of 
effort would be different still. In this situation, the curve is more shallow at Joe's position above the goal, and we wouldn't 
expect as much value for effort at each quiz. 

Together, these theories suggest that challenging goals are more motivating than easy goals, but that if a goal is 
judged to be simply too far away motivation will be poor. Also, motivation (because of the value of effort) will be less if we 
tell students, for example, that they start a class with 100 points and are working to avoid losing points than if they start a 
class with 0 points and are working to gain points. 

However, both of the above theories look only at objective performance as the outcome on which goals are based, 
and this is potentially a problem if we want to apply these theories to instruction. There is a large body of literature within 
social and educational psychology suggesting that learners bring certain goal orientations with them to the instructional 
setting (e.g. Dweck & Leggett, 1988; see Marsh, Craven, Hinkley, & Debus, 2003; and Murphy and Alexander, 2000, for 
reviews), that many learners are motivated by internal learning goals (involving a personal perception of mastery) while 
others are motivated by externally-defined performance goals, and that these learning goals are really the more adaptive and 
desirable of the two. When considering how goal orientations might interact with goal messaging manipulations in an 
instructional context, we would expect that learners with a performance goal orientation would be more strongly influenced 
than those with a learning goal orientation. 

Predictions 

In order to investigate the above, goal message framing and goal difficulty were manipulated within a Web-based 
learning task. Half the learners were told that they started the learning task with no points, but would gain points for each 
question answered correctly on an objective knowledge assessment. The other half were told that they started the learning 
task with a perfect score, but would lose points for each question answered incorrectly on the objective knowledge 
assessment. In addition, each learner was assigned one of three performance (as a final point total) goals on the objective 
knowledge assessment - one easy, one challenging, and one extremely difficult. Finally, all learners were administered a 
goal orientation inventory in order to assess the impact of goal orientation on goal messaging framing and goal difficulty. 

It was expected that learners beginning with no points and working to gain points would score higher on the 
objective knowledge assessment than learners beginning with a perfect score and working to avoid losing points. Likewise, 
it was expected that learners assigned to the challenging goal condition would score higher than learners with an easy goal 
or those with an extremely difficult goal. Also, it was expected that learners with a challenging goal, and working to gain 
points, would obtain the higher scores than learners in the other five groups. Finally, it was expected that learners with a 
stronger performance goal orientation would be more strongly influenced by the goal messaging manipulations than those 
with a learning goal orientation. 

Method 

Participants 

Participants were 141 female and 79 male undergraduate students enrolled in introductory computer and 
introductory educational technology classes at a large public university and a large community college (mean age=21). 
Design 

Participants were randomly assigned to one of six treatment groups representing Message Frame (above goal vs. 
below goal) and Goal Difficulty (easy vs. challenging vs. highly improbable) between subjects factors. These factors were 
crossed with 10 Tutorial Sections as a repeated measure, producing a 2 Message Frame X 3 Goal Difficulty X 10 Tutorial 
Section design. The treatment conditions are defined as follows: 

Message Frame. Participants in the above goal group were shown an initial performance score of 100 points, and 
told that for each question missed they would lose 1 point. All feedback messages were displayed from this perspective, i.e. 
“You currently have 94 points. You lost 1 point on this question.” Participants in the below-goal group were shown an initial 
performance score of zero points, and told that for each question they would gain 1 point. 

Goal Difficulty. Goal Difficulty groups were created as follows: an “easy” group, where the goal was to score at 
least 20% correct (achieved by 90% of participants in a pilot study, see Anonymous Author, 2007, Appendix B); a 
“challenging” group, where the goal was to score at least 79% correct (achieved by 10% of participants in a pilot study), and 
a “highly improbable” group, where the goal was to score 92% correct (reached by no participants in a pilot study). Goal 
Difficulty was stated in terms of final points needed to meet the goal, i.e. “Your goal is to have 79 points at the end of the 
tutorial.” The criterion for the “challenging” group condition was taken from Latham and Seijts (1999), and Knight, 

Durham, and Locke (2001), who use this criterion to set similar goal conditions for workplace tasks. 

Dependent measures. Section quiz scores served as the main dependent measure. Participants also completed items 
designed to assess a) intrinsic motivation, b) tutorial difficulty, and c) perceived amount learned. Following the completion 
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of the experimental sessions, the data were analyzed in a 2 Message Frame x 3 Goal Difficulty between-participants 
factorial design, with AGI subscale scores serving as regressors. 

Materials 

A History of the Internet Web-based tutorial was used as the experimental stimulus (see Anonymous Author, 2007, 
Appendix B for a discussion of the development and psychometric properties of this instrument). The tutorial consisted of 
10 sections, each on a different topic related to the history of the Internet. Each section began with a two-page text passage 
followed by a 10-item multiple choice quiz. 

Participants completed the outcome goal, ability goal, learning goal, and normative goal subscale items of the 
Achievement Goal Inventory (AGI) developed by Grant and Dweck (2003). 

In addition, participants were asked to rate their agreement with several statements on a four-point Likert-type 
scale. The first four were intrinsic motivation items (from Elliot & Church, 1997; Grant & Dweck, 2003): a) I thought the 
tutorial was very interesting, b) the tutorial was a waste of time, c) the tutorial was boring, and d) I enjoyed the tutorial very 
much. Two difficulty items: a) the tutorial was difficult, and b) the tutorial was challenging. Last was the statement I 
learned a lot from the tutorial. 

Each participant’s age, gender, years in college, and major were also collected. 

Procedure 

All materials were administered during a single, regularly-scheduled class meeting. Students were briefed on the 
nature and purpose of the tutorial and study, and given brief instructions on how to proceed. These instructions included the 
explicit statement, repeated at least twice verbally, that all responses were completely anonymous, that the only information 
given to the course instructor would be a list of students who completed the entire tutorial, and that extra credit was only 
contingent upon completing the tutorial and not on score. 

Eollowing these verbal instructions, students were instructed to open a Web browser and enter the URL of the 
tutorial. After an initial page verified the correct browser settings, the participant continued to a page of written instructions 
that re-stated the verbal instructions given previously and informed the participant of the nature of the task. Next, the Web 
application randomly assigned the participant to one of the six treatment groups, and a page was presented to collect the 
demographic data. 

Eollowing this the AGI was administered. Eollowing the AGI, the participant was presented with a goal message 
and framing page, on which goal message statements appropriate to the participant’s treatment group were displayed. 
Students in the below-goal frame, and “challenging” difficulty group, for example, were presented with the following 
message: 

You start the tutorial with 0 points. You will gain 1 point for each question you answer correctly. 

Your goal should be to have a score of 79 points or more at the end of the quiz. 

The participant then clicked a button and continued to the start of the first tutorial section. Each tutorial section 
began with a two-page text passage, followed by 10 multiple choice questions based on the text passage. Each question was 
displayed on a page by itself, and was presented in a random order. Students could navigate back and forth between the two 
text pages, but could not return to the text once the quiz was started. Also, once a response is submitted the student could 
not return to the question, and each question required a response before the next question could be accessed. 

On each page of the text passage, the overall score was displayed in the upper right comer of the browser 
window on the two text pages of each section. In the upper left corner, the Goal Difficulty message was displayed. 

Eor a participant in the above-goal frame, and “highly improbable” difficulty group, the message Score at least 92 
points overall! appeared in large bold text in the upper left, and the message (assuming a single question had been 
missed) Overall score thus far: 98 point. Your've lost 2 points thus far. was presented in the upper right. 

Eollowing the two text pages at the beginning of each section, the participant was presented with each multiple 
choice question. As in the two text pages at the beginning of each section, the Goal Difficulty message was presented in the 
upper left comer of the browser window, while the overall score and section score thus far were displayed in the upper right 
corner, stated in a manner consistent with the goal frame group. 

Once the participant submitted a response to a quiz item, a feedback screen would appear for that item, indicating 
the correct answer, along with updated section and overal scores. Eollowing the last quiz item and feedback page in each 
section, a section summary page was presented which re-stated the overall goal, gave the percentage correct on the section, 
the current score, and indicated the number of points lost or gained in the section and overall. Eollowing the section 
summary, the next tutorial section begins. 

Once the section summary of the tenth and final tutorial section was displayed, participants were prompted for the 
intrinsic motivation, tutorial difficulty, and perceived learning item ratings. After the student completed the final item, they 

were thanked for their participation and dismissed from class. 
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Results and Discussion 

Group differences in section quiz scores were analyzed in a general linear model analysis of covariance (GLM 
ANCOVA), with Tutorial Section as a repeated measure and scores on each of the AGI scales as covariates. Mauchly’s test 
of sphericity on the within subjects effect of Tutorial Section was significant, thus a Huynh-Feldt epsilon value of .982 was 
used to adjust the degrees of freedom for all F-tests in the ANCOVA. 

A main effect was found for AGI Ability scale score, F (1,148) = 4.87, p=.029, r|p^=.032. Generally, as participants 
scoring higher on the AGI Ability scale achieved lower section quiz scores. While the main effect was significant, strength 
of association as measured by indicates a weak relationship. 

While no main effects were found for the treatment variables, a significant interaction was found between Goal 
Difficulty and AGI Ability scale score, F (2, 148)=2.73, p=..029, Pp^=.036. Although the strength of association is weak, 
participants in the challenging and improbable Goal Difficulty groups with higher AGI Ability scores tended to achieve a 
lower mean section quiz score. For participants in the easy Goal Difficulty group there was no apparent relationship 
between AGI Ability scores and mean section quiz scores. 

A significant yet small interaction was found between Tutorial Section and AGI Ability scale score, F (8.842, 
1308.649) =1.94, p=.044, Pp^=.013. While participants scoring lower on the AGI Ability scale tended to perform better on 
section quiz scores overall, this effect was not consistent across Tutorial Sections. 

A significant yet small interaction was found between Tutorial Section, goal Message Frame, and AGI Ability 
score, F (8.842, 1308.649) = 1.93, p = .046, Pp^=.013. As the tutorial progressed, participants in the below goal group who 
scored higher on the AGI Ability scale tended to obtain lower scores on the tutorial section quizzes, while the opposite was 
true of those in the above goal group. Interestingly, this effect peaked at the midpoint of the tutorial, and then began to 
diminish. Again, although the interaction was significant, strength of association was found to be quite weak. See Figure 2. 



Figure 2. Regression of AGI Ability score on section quiz scores for each level of goal Message 
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Figure 3. Mean section quiz scores by treatment group. 
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Similar to the above interaction, a significant Tutorial Section X Goal Difficulty X AGI Ability score interaction 
was found, F (17.684, 1308.649) = 1.84, /7=.018, r|p^=.024. Students with a higher AGI Ability score in the improbable Goal 
Difficulty condition tended to score poorer on most, but not all, section quizzes. The opposite was true for participants in the 
easy goal condition, but only on the last two sections. Patterns for participants in the challenging goal condition tended to 
follow those for the improbable goal condition but have lesser slope. Again, strength of association was found to be weak. 

In addition to the Goal Difficulty X Tutorial Section interaction with AGI Ability score presented above, a Goal 
Difficulty X Tutorial Section X AGI Learning score interaction was found, F (17.684, 1308.649) = 1.84, p=.018, r|p^=.024. 
Figure 4 displays a graph of the regression of AGI Learning scale scores on each section quiz score for each of the Goal 
Difficulty groups. In the first section, a small, positive relationship appears to exist between AGI Learning score and section 
quiz score for participants in the easy goal condition. Interestingly, as the tutorial progresses this relationship reverses with 
increasing slope until it reaches its maximum in section five, at which point the slope decreases until the relationship 
reverses once more in sections nine and ten. A nearly reverse pattern is seen for participants in the improbable Goal 
Difficulty condition. For participants in the challenging goal condition the relationship between AGI Learning score and 
section quiz scores tends to be either slightly negative or flat. As with the findings of significant effects reported previously, 
it is important to note that strength of association as measured by Pp^ was found to be small. See Figure 3. 

Thus, the present study found that the effects of Message Frame and Goal Difficulty, when they occurred, were 
dependent on the goal orientation of the learner, and were generally small. Also, with the exception of Goal Difficulty, the 
ordinal position of a given quiz in the overall tutorial determined whether a goal message would have an impact at all, or 
the direction that impact would take. 
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Face-to-Face Support for Distance Learners in a Mega University 
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Anadolu University, Turkey 

Abstract 

This paper is about one of the most crucial aspects of an open and distance learning: student support. It 
intends to reveal the results of an evaluation study in which perceptions of the learners about the face-to-face 
support service were investigated. The paper provides not only results of a study but also introduces a face-to-face 
student support model for large number of distance learners. So, it might be beneficial for the academicians who are 
interested in management of open and distance learning. 



Introduction 

Although controversies over the effectiveness of open and distance learning continue, the number of open 
and distance learning providers has increased dramatically over the last decade. Today, around 30 open universities 
and countless of educational institutions and other organizations operating in not only developed but also in 
emerging regions of the world have launched or started to think of launching open and distance learning programs 
mainly in order to be able to reach more students with fewer expenses (Duffy & Kirkley, 2004). That is why, open 
and distance learning is considered as “the fastest growing sector in education worldwide” (Simpson, 2002). 

This paper is about one of the most crucial aspects of a successful open and distance learning initiative: 
student support. Experts agree that although student support is one of the crucial elements of any kinds of a 
successful open and distance learning initiative, it has not been got the attention it has deserved (Simpson, 2002; 
Tait, 2000). However, advances in information and communication technologies have provided new opportunities to 
support distance learners (Scheer & Lockee, 2003). For example, the British Open University has started to use 
online technologies such as email, chat, and discussion boards, so forth to help its learner in 1995. Studies on those 
attempts have shown that facilitators’ active interactions with students have significant effects on the quality of 
online distance learning (Thomas, Carswell, Petre, Price, 1998). Thus, success and quality of a distance education 
program heavily relies on establishment of an effectively working support system (Mills & Ross, 1993). 

Although every open and distance learning system provides some sorts of student support service, majority 
of these are found extremely ineffective and underdeveloped (Peters, 1998). These kinds of services sabotage open 
and distance learning initiatives (Chute, Thompson & Hancock, 1999). That is why most of the accreditation 
institutions (e.g. The Commission on Institutions of Higher Education, The Distance Education and Training 
Council) look for quality of the student supports services when evaluating open and distance learning provider 
candidates. 

Experts (e.g. Carnwell & Harrington, 2001; Hyland, 2001; Sheer & Lockee, 2003) emphasize importance 
of fulfilling students’ needs to ensure the quality of student support. According to Tait (2000), a student support 
service has three primary functions for this fulfillment: Cognitive, affective and systemic. Cognitive function deals 
with the support and development of learning while affective is related to the creation of an environment that 
enhances commitment and self-esteem. Systemic function is about administrative processes. Similarly Berge (1995) 
states that online learners need four types of supports: Pedagogical, social, managerial, and technical. Although 
Berge considers these supports for online learners, they can easily be related to any kind of distance learning. 
Pedagogical support refers to the interactions between instructor and students on content and/or task oriented topics. 
Social support includes the creation of an environment where online (distance) students meets with other students or 
instructors or others and interact on usually non-academic (or non task-related) topics. So the feeling of isolation 
diminishes and a feeling of socially presence increases. Managerial support is related to the organizational and 
procedural issues such as registration while technical support requires solving learners’ technical problems. All sorts 
of open and distance learning programs try to offer these or similar type of student support. 

There are various means to provide required support to distance learners. Same place-same time support, or 
face-to-face meeting is one of these means. Supporting students through face-to-face meetings in remote locations 
has been into consideration for years. This way of support can be regarded as the ancestors of blended learning. 
Activities that learners and support staff realize during the meetings: working on projects, drill and practice, lecture 
(Simpson, 2002). 
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Anadolu University of Turkey, as one of the important open and distance learning provider of the country, 
has been supporting its distance students via various support systems including face-to-face meetings since its early 
days, in which mainly lecture method has been adopted. 

Anadolu University serves students in Turkey, Turkish communities in the European Union and Northern 
Cyprus. It currently has 24,300 on-campus students and around 1,150,000 off-campus distance students. The recent 
figures reveal the role of Anadolu University in Turkish Higher Education System; 336,000 distance students who 
enrolled in 2005/06 after the centralized university entrance exam constitute over 40% of all university students and 
99% of all distance education students in Turkey. The majority of its programs offer four-years long undergraduate 
and two-year long associate degrees. The instructional strategy is primarily print-based and requires self-study. In 
other words, students are expected to study their textbooks at their own pace, alone, and to take scheduled 
centralized exams administered at remote locations. The self-study is also supported with several services including 
broadcast television programs aired by a state channel throughout the country, video and radio programs distributed 
on cassettes, CDs or DVDs, remote evening classes, and computer-supported learning environments. The rationale 
behind this sort of an instructional approach is common to all open and distance learning initiatives in emerging 
countries. These are based on (1) outreach to as many learners as possible in cost effective ways, and (2) providing 
alternatives for learners’ limited access to the other technologies including VCRs, computers and even television 
broadcasts. 

Eor certain courses, pedagogical support is provided via face-to-face lecture sessions. The University has 
agreements with local universities (currently 41 universities) in 68 cities out of 81 of the country to hire their 
personnel and facilities to offer these lectures to the distance learners. The lectures are given during the evenings 
(after work hours) and weekends. Every year approximately 25.000 learners regularly attend those lectures. 

Academic tutoring courses which are used as a learner support in Anadolu University open and distance 
education programs are the only face-to-face environment in which learner-instructor interaction is at the highest 
level. With the help of academic tutoring courses, distance learners can ask the instructor about the content they 
haven’t understood and get answers, have the opportunity to reinforce what they have learnt, get rid of the sense of 
isolation and acquire a sense of belonging by establishing in-class interaction with other learners. Moreover, 
distance learners view this environment as a place in which they strive for learning and feel responsible for it. Eace- 
to-face academic tutoring services have a significant role in the distance education practice carried out in Turkey. 
Nevertheless, as there isn’t such a face-to-face academic tutoring service as ours in the distance education practices 
of other countries, no research finding related to this environment and the attitudes of learners making use of this 
environment has been found. 

Therefore, the purpose of this study is to analyze who the distance learners attending face-to-face courses 
are, in other words to analyze the learners’ demographic characteristics, and to find out why they prefer to attend 
face-to-face courses although they are distance learners and they are supported through various services-to find out 
their attitudes towards learning environments 



Methodology 



A survey consisting of six questions designed to find out the demographic characteristics of the distance 
learners attending face-to-face academic tutoring courses and the frequency of their attendance in such courses (see 
table 1), and a 5-point Likert type scale questionaire with 9 items designed to find out why these learners make use 
of this service (see table 2) was conducted on 520 learrners attending face-to-face academic tutoring courses in 
Eski^ehir. After eliminating forms that were incomplete or missing many items, the sample of the study consisted of 
480 learners. Administration of survey were completed one month before the semester ended on April, under the 
supervision of the researchers. 



Data Analysis 

Various statistical procedures were utilized in the study with the p value of .05 accepted as the level of 
meaningfulness. In addition to investigating means, the reliability of the items were tested by Cronbach Alpha 
Analysis. Moreover, the components of the questionnaire items were analyzed by factor analysis and it was found 
that there are two components and that both of their summability are tested. However, as the correlation between 
them was high, they were regarded as one component. Also, the differences between the reasons for these distance 
learners’ attending academic tutoring courses (in relation to their demographic characteristics) were tested via t-test 
and anova test. Statistical procedures were handled by using SPSS 14.0 and the results are interpreted at the end of 
the paper. 
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To assess why these distance learners attend academic tutoring (face-to-face) courses, a 5-point Likert type 
scale was used. A scale with 9 items was developed by the researchers (see table 2). The scale reliability was found 
to be 0,806. 



Findings 

When the demographic characteristics of 480 learners making use of academic tutoring courses and taking 
the questionnaire are analyzed, it is found that women (54.2%) benefit from this service more than men (45.8%) do, 
that the majority of learners are single (97,1%) and that they are learners who are enrolled for undergraduate 
programs (91,9%). When it is analyzed in terms of classroom variable, it is found that mostly first year students 
(73,5%) benefit from academic tutoring courses and the majority of learners (78,5%) do not have a job. It is also 
found that, in terms of the location where they live, 56,5% of the learners live in the city center where the academic 
tutoring service is provided and take these courses in the city in which they live, that 23,9% take these courses by 
coming from district-subdistrict-village of the city where this service is provided and that 19,6% take them by 
coming from another city center. When the attendance to academic tutoring is questioned in terms of frequency, the 
majority of learners (82,9%) report that they attend these courses regularly. 



Table 1. The Distance Learners Attending Academic Tutoring Courses: 
Their Demographic Characteristics and Frequency of Attendance 







Frequency 


Percent 




Female 


260 


54,2 


Gender 




Male 


220 


45,8 




Single 


466 


97,1 


Marital Status 


Married 


10 


2,6 




Divorced 


4 


0,8 




Undergraduate 


441 


91,9 


Type of program 




Two-Year Degree 


39 


8,1 




First year student 


353 


73,5 


Grade 


Second year student 


66 


13,8 




Third year student 


61 


12,7 




I work 


103 


21,5 


Work status 




I don’t work 


377 


78,5 




In the city center where academic 

tutoring service is provided 

In a district-subdistrict-village of the 


271 


56,5 


In what kind of a place do you live? 


city where academic tutoring service is 
provided 


115 


23,9 




In another city near the one where 
academic tutoring service is provided 


94 


19,6 




I attend regularly 


398 


82,9 


How often do vou attend academic 


tutoring courses (face-to-face lessons)? 


I attend when I have the opportunity 


71 


14,8 




I attend only before the exams 


11 


2,3 


Total 




480 


100,0 



66 





The appropriacy of the variables in the scale for the factor analysis was tested and it was mentioned that the 
factor analysis in which values KMO=0,818 and p=0,0001 <0,05 had been found was significant. As a result of the 
factor analysis carried out, two components (Q1-Q3 and Q4-Q9) were found and as the correlation between these 
components was 0,589. As the correlation was high, these two components were regarded as one. 



Table 2. Factor Analysis for Determining Why Distance Learners Attend in Face-to-Face 
Academic Tutoring Courses 



Descriptive Statistics 


Extraction 


Mean 


Std. 


Total explained 
of variance 


I believe that these face-to-face courses will 
increase my success in exams. (Ql) 


,610 


3,95 


1,197 




Getting together with the other students makes 
me learn more easily. (Q2) 


,511 


3,44 


1,265 




I feel like a student when I attend face-to-face 
courses. (Q3) 


,546 


4,10 


1,177 




I feel more comfortable when I learn a topic in a 
face-to-face course. (Q4) 


,635 


4,16 


1,276 




I prefer attending face-to-face courses to self- 
study. (Q5) 


,570 


3,63 


1,571 


56,223 


It is easier for me to remember what I’ve learnt in 
a face-to-face course. (Q6) 


,743 


4,34 


1,139 




Attending face-to-face courses makes me learn 
more easily. (Q7) 


,789 


4,40 


1,053 




I think that face-to-face learning is more effective 
than self-learning. (Q8) 


,652 


4,35 


1,102 




I think that face-to-face learning takes less time 
than self-study does. (Q9) 


,504 


3,89 


1,303 




Extraction Method; Principal Component Analysis. 



The nine items in 5-point Likert type scale questionnaire about the distance learners’ attendance in 
academic tutoring courses were taken as one dimension, and the average was calculated. According to this, the 
learners' gender and marital status, type of program they are enrolled in, their grade and work status, the place they 
live in, and their attendance frequency were tested via t-test and anova test. These results have been displayed in 
table 3. 



Table 3. The differences related to learners' demographic characteristics and the frequency of their 



attendance in academic tutoring courses 







Mean 


Std 




Gender 


Eemale 


4,0842 


,75617 


t-test t= 1,699 

P=0,09>0,05 


Male 


3,9636 


,79567 


Marital Status 


Single 


4,0315 


,77164 


f-test f=0,892 
P=0,41>0,05 


Married 


4,1111 


,68893 


Other 


3,5278 


1,44409 


Type of program 


Undergraduate 


4,0421 


,76589 


t-test t= 1,248 
P=0,213>0,05 


Two-Year Degree 


3,8803 


,88022 


Grade 


Eirst year student 


3,9355 


,79367 


f-test f=10,174 
P=0,0001<0,05 


Second year student 


4,2559 


,68520 


Third year student 


4,3242 


,63806 
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Work status 


I work 


4,0982 


,74784 


t-test t= 1,022 
P=0,307>0,05 


I don’t work 


4,0100 


,78348 


In what kind of a 
place do you live? 


In the city center where academic tutoring 
service is provided 


4,1037 


,73345 


f-test f=3,016 
f=0,049<0,05 


In a district-subdistrict-village of the city 
where academic tutoring service is 
provided 


3,9536 


,81090 


In another city near the one where 
academic tutoring service is provided 


3,9054 


,83386 


How often do you 
attend academic 
tutoring courses 
(face-to-face 
lessons)? 


I attend regularly 


4,0826 


,73753 


f-test f=6,847 
P=0,01<0,05 


I attend when I have the opportunity 


3,8185 


,92300 


I attend only before the exams 


3,4444 


,69567 



The significance of the difference between the scores of reasons for learners’ attending face-to-face courses 
and their gender, type of program they are enrolled in, and their work status was analyzed via t-test. The significance 
of the difference between the learners’ marital status, grades, the location they live, and their frequency of 
attendance and the scores of reasons for their attending face-to-face courses was analyzed via anova test (f-test). 
There is no significant difference found between the learners’ gender, marital status, type of program they are 
enrolled in, and their work status and the scores of reasons for their attending face-to-face courses. 

However, with respect to attending face-to-face courses, that women have higher average scores than men 
do, that married students have higher average scores than the single and the other learners do, that undergraduate 
students have higher average scores than two-year degree students do, and that the working learners have higher 
average scores than the non-working ones do indicate that these learners have a more positive attitude towards 
academic tutoring courses and that they have more belief in these courses. 

The difference between the learners’ grades and the scores of reasons for their attending face-to-face 
courses has been found significant. According to this, third year students have a higher average score than first year 
and second year ones do, and second year students have a higher average score than first year students do. There is 
no significant difference between second and third year students; however, these students have higher scores of 
reasons for attending face-to-face courses, and this difference has been found significant. This shows that as the 
learners’ grades go up, they become more conscious about attending academic tutoring courses and that they 
develop more positive attitudes towards such courses. 

The significance of the difference between type of place the learners live in and the scores of reasons for 
their attending face-to-face courses was tested via anova, and the difference has been found significant. There is a 
significant difference between the learners who live in the city center, where face-to-face-courses are held, and the 
ones who come from different cities to attend face-to-face courses. 

There is also a significant difference between the learners' frequency of attendance and the average scores 
of reasons for their attending face-to-face courses. That the students who attend face-to-face courses regularly have a 
higher average score than those who attend these courses when they have the opportunity and the ones who attend 
them only before the exams show that regular attendants have a more positive attitude towards academic tutoring 
courses. Although those who attend these classes when they have the opportunity have higher scores than the ones 
who attend only before the exams, this difference has not been found significant statistically. 

The correlation between the learners’ attendance frequency in academic tutoring face-to-face courses and 
their marital status, grade, and work status was tested via Pearson Chi-Square test. As Chi-Square, which was 
calculated as a result of Excat Monte Carlo test, was p=0,371>0,05, that there is no meaningful correlation between 
learners’ attendance frequency and their marital status is observed. 

Again after the Excat Monte Carlo test, there is no meaningful correlation found between the learners’ 
attendance frequency and their grades (p=0,399>0,05). 
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As the correlation between the learners’ attendance frequency and their work was Chi-Square 
p=0,002>0,05, it has been found meaningful. 

Lastly, since the correlation between the learners’ attendance frequency and the place they live in was Chi- 
Square p=0,787>0,05, there is no significant relationship found between them. 



Conclusion 

Academic tutoring courses were integrated into the system as an important component in the 1982-83 
school year, when they started to be applied in the Open and Distance Education programs of Anadolu University. 
While different communication technologies and media, especially computers and the support systems based on the 
Internet were made use of in the development of university’s open and distance education system, academic tutoring 
courses which are carried out in the traditional classroom environments maintain their importance in the system as a 
support component. This seems to be against the nature of distance education, because this educational approach has 
a perception representing the learner and the instructor’s temporal and spatial separateness that is their distance. For 
this reason, distance learners’ desire for maintaining their learning through face-to-face environments that make 
them dependent on the time and place as well as the technologies and environments based on individual learning 
which provide the learning support at the time and place they themselves want make it compulsary to investigate the 
academic tutoring service. From this point of view, in the study, why the learners make use of this service provided 
in a face-to-face environment, what kind of demographic characteristics they have, and whether their reasons for 
attending face-to-face courses change according to their demographic characteristics were questioned. 

In the open and distance education approach emphasizing individual learning, learners’ desire to make use 
of the tutoring courses can be associated with the learning culture existent in Turkish education system. For various 
reasons, Turkish learning culture has a structure in which instructor rather than the learner is the determinant, 
teaching rather than learning is the focus and there is a one way interaction from instructor to learner (Alkan, 1997). 
Owing to this fact, that distance learners want to attend academic tutoring courses, the ones which serve the learners 
with a traditional understanding of education, is an expected situation. 

Morever, as Turkey is a country of young population, there is a great number of students wishing to have a 
university education. As a result of this fact, students are admitted to university programs via a centralized 
examination system. Those students who have high levels of success and effective studying skills firstly prefer face- 
to-face university education rather than distance programs. This situation results in the fact that those students who 
lack systematic self-study skills prefer academic tutoring courses, where the tutor is the determinant factor, as the 
supporting service. 

The demographic data collected in this study can be assesed in different ways. It can be found that women 
attend more than men do and those attending the lessons are mostly single and enrolled for an undergraduate 
program (Table 1). Considering that academic tutoring courses are chosen among the courses existent in the 
curriculum of undergraduate programs, it can be said that the participation of two-year degree learners is lower as 
they can take only these mutual courses existent in their programs. Furthermore, the fact that the participation is 
considerably high for the first year classes is another striking finding. One of the reasons for this may be that 
learners who are accustomed to the traditional educational style do not know this system based on individual 
learning very well when they start distance education. 

Another finding of the study is that the majority of learners attending the academic tutoring courses 
(78,5%) do not have a job. It is known that the demand for university education in Turkey has increased only for 
having a degree as well as personal growth and career development. Related to these reasons mentioned, the fact 
that formal university education doesn’t have the necessary physical capacity to meet this demand causes the 
teenagers having just graduated from high school to prefer open and distance education. As a result of this, it is 
possible to say that people making use of academic tutoring courses consist of young people who have not started 
working yet. 

Academic tutoring courses are carried out successfully in 74 city centers. What is crucial in choosing these 
city centers is whether there is a faculty or college where tutoring lessons can be given and whether or not there are 
enough instructors for this in that city. In cities where the adequate resources are provided, academic tutoring 
courses are started immediately. In the study, it has been found that 56,5% of the learners can make use of this 
service in the cities they live in, 23,9% get this service by coming from other districts or villages near this city and 
the remaining 19,6% come from another city to benefit from this service. 
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Overall, this study reveals the demographic characteristics of distance learners attending in face-to-face 
academic tutoring courses and analyzes the reasons for their attendance in these courses in accordance with their 
demographic characteristics. However, a generalization of the results in this study can be misleading. To do similar 
studies about academic tutoring courses in different cities and comparing the results of those studies are going to 
make it possible to develop more effective and productive academic tutoring courses. 
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Abstract 

Class sizes in lower division engineering are getting larger, as is the diversity of learning needs to be 
addressed by the instructors. Computer-mediated instructional tools and especially online tools have proven helpful 
to faculty in reaching larger numbers of students both in traditional (Romiszowski & Mason, 2004) and blended- 
learning environments (Picciano & Dziuban, 2007). However, the process of integrating these tools in classroom 
activities is not trivial task. When looking at the instructional process as a mediated activity, tools define how the 
participants engage in classroom tasks to sustain meaningful learning (Barab et al., 2004; Engestrom, 1999). The 
mediating role of online tools on the structure of the instructional process generates both contradictions and 
synergies (Jonassen, 2000). Both instructors and students are attracted by the flexibility and convenience of online 
tools, but tend to resist them because they seem very impersonal, tend to dilute the control over the instructional 
activities, and are subject to distracters that negatively impact the learning process. 

In this context, a Mechanics of Material course with an enrollment of 80 to 150 on-campus students was 
offered with two enrollment options for lectures: (a) face-to-face attendance; (b) online participation either live or 
delayed (archived lectures). Students were free to choose any combination of the options for viewing the class. The 
Webex® online conferencing tool was used for the synchronous online delivery of the lectures as alternative to the 
face-to-face participation in the course. 

Due to the novelty of this approach for both instructor and students, the research explored: a) how the 
inclusion of Webex impacted students’ behavior; b) how these changes impacted classroom performance; and c) 
students’ attitudes toward the online conferencing tool. 

Introduction 

Class sizes in lower division engineering are getting larger, and in this context, the diversity of learning 
needs that instructors need to address increased significantly. Computer-based instructional tools and especially the 
web-based ones helped lately faculty to reach a larger numbers of students both in traditional (Romiszowski & 
Mason, 2004) and in blended-instruction settings (Picciano & Dziuban, 2007). However, integrating these tools in 
classroom activities is not trivial task. When looking at the instructional process as a mediated activity, tools define 
how the participants engage in classroom tasks to sustain meaningful learning (Barab et al., 2004; Engestrom, 
1999). The mediating role of online tools on the nature and the structure of instructional process generate both 
contradictions and synergies (Jonassen, 2000). Elexibility and convenience of online tools is appealing to both 
instructors and students but both groups tend to resist them. Instructors tend to resist using online instructional tools 
mainly because they seem very impersonal. 

Instructors also feel they lose control over the instructional process when using these tools. The personal 
contact with students in face-to-face classroom settings is important for instructors because it allows them to: (a) 
build a community with a certain level of trust; and (b) get quick feedback related to instructional issues and then 
promptly react to avoid anxiety and complaints from students. To achieve a similar level of contact in web-based 
teaching the instructor needs to engage in more planning and active monitoring. The instructor will also need to 
master in a relatively short time those skills associated with the effective use of online instructional tools. On the 
other hand, online instructional tools when properly integrated in the instructional process can create a synergy that 
facilitates and enhances the communication process between the instructor and large groups of students. This 
synergy can also help the instructor engage more students in informal activities such as review sessions and office 
hours"*. Eor example, the synchronous online instructional tools available today offer voice and video tools that 
reduce the gaps between the face-to-face and online instruction. In addition, both students and instructors can use 
recordings of live instructional sessions for further review. 

Eor most students the attractive features of online instructional tools are their flexibility and convenience. 
Not being bound to be at a given location and/or time allows for more flexibility in personal and professional time 
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management. Students can also enroll in courses that have a scheduling conflict if taken in face-to-face environment 
and then avoid potential delays in the completion of their degree. 



In addition, students benefiting of online tools can participate more freely in extra curricular activities, job 
interviews, take care of personal and family matters, and miss classes when they need to miss. Some students tend 
to resist online instructional tools because personal interaction with the instructor, the motivator that helps keep 
them engaged in the course, is missing. Another factor that generates student resistance to online tools is the lack of 
scheduling structure. Because quite often this factor determines the success of task completion, a flexible schedule 
requires students to build the self-discipline of watching the lectures and keeping up with the coursework in a timely 
manner. Students engaged in online activities have to deal with distractions, such as television and friends, when 
watching the lectures online. These distractions are not present in a face-to-face lecture, which makes it easier for 
students to focus on the material presented. To summarize, the primary advantages of online instruction are 
flexibility and convenience of being able to view the lectures, and review instructional sessions from any location 
that has an internet connection. The primary disadvantage of online instruction is the lack of structure specific for 
the face-to-face classroom. Some students need the structure of a face-to-face lecture to motivate them to keep up 
with the course material. 

Instructional Context 

A Mechanics of Material course with an enrollment ranging from 80 to 150 on-campus students was 
offered with three enrollment options for lectures; (a) face-to-face attendance in classroom; (b) online live 
participation; and (c) delayed (recorded) via streamed recorded lectures. Students were free to choose any 
combination of the options for viewing the class. The instructional tool introduced to support the described format of 
this course was Webex" , an online conferencing tool was used for the synchronous online delivery of the lectures 
as alternative to the face-to-face participation in the course. With a tablet PC and a wireless microphone the 
instructors was able to simultaneously project his tablet screen and his voice in the classroom, to stream the same 
information for synchronous engagement, and to record it for asynchronous engagement. What made Webex a good 
match for the instructional process was the nature of the course material based on a rather small pool of concepts 
applied in a large set of contexts through extensive worked examples (Figure 1). 
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Figure 1. Webex screenshot for a sample typical worked example part of the lecture activities. 



This tool also allowed the instructor to, on one hand engage live students through a chat space and, on the 
other hand to tape lectures and make them available to all students to be used for review or as replacement of 
synchronous participation in the course. Webex was also the tool used by the instructor to conduct the review 
sessions and office hour sessions for the course that significantly increased students’ participation in this type of 
remedial instructional activities. 

The complexity generated by the integration of this type of online instructional tool in classroom settings 
requires a long-term commitment from the instructor, to allow for an in-depth analysis of the impact on the learning 
process. This paper reports on the first stage of this commitment. Due to the novelty of this approach for both the 
instructor and students, the overall research questions were exploratory in nature: How did the inclusion of the 
online instructional tool impacted students’ behavior? What is the impact of this tool on student performance? What 
is the students’ perception of these tools? 



Methods and Methodologies 

This study followed Spring Semester of 2007 (SP07) implementation of Webex as online instructional tool. 
Of the 88 students enrolled during the SP07 semester, 76 returned complete data, and after eliminating outliers a 
sample of 74 students was retained for this study. Scores for homework, final exam, and final grade were collected 
from Blackboard and transformed into percentages of maximal score. A survey administered by email collected the 
behavioral and attitudinal measures. Extra points rewarded those students’ returning the survey. 

Attendance behavior was the self-reported behavioral measure calculated as the weighted sum of students’ 
self-reported attendance behavior. Students were asked to report percentages of their use of each of the three modes 
of attending the lecture: face-to-face, Webex live, and Webex archived. Because the reported percentages for the 
three attendance modes had to add up to 100 percent they were used as weights to convert the type of attendance 
behavior in a continuous variable. The three types of attendance behavior were scored based on the level of 
commitment required from the student as follows: 5 for face-to-face attendance (requires time and space 
commitment), 3 for Webex live (requires only time commitment), and 1 for using the Webex taped lectures as it 
requires the least level of commitment. The result was a continuous variable ranging from 1 for 100% Webex taped 
lectures, to 5 for 100% face-to-face lectures. 
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To measure students’ attitude toward the online tool the survey used three major measures. First five 
questions adopted from Siau et al. (2006) measured students’ perception of interactivity afforded by the online tool 
for class discussions, involvement in the class activities, and feedback from instructor. The next two sets of three 
questions, adopted from Technology Acceptance Model (Davis et ah, 1989), targeted students’ perceived ease of use 
and respectively perceived usefulness associated with the use of Webex. All scales used a 9-point Likert scale with 1 
representing Strongly Disagree and 9 representing Strongly Agree. The Cronbach’s alpha coefficient for interactivity 
level is .832, for perceived ease of use .909, and respectively for perceived usefulness of Webex as instructional tool 
was .922. The overall reliability of the instrument as measured wit the Cronbach’s alpha coefficient was .908. The 
Cronbach’s alpha coefficients both for individual scales and for the instrument as a whole exceed Nunnally’s (1978) 
threshold of .70 and then suggest a high reliability of the instrument and its components. 

Design and Results 

The exploratory question for this part of the study was: Did the nature of attendance behavior students had 
chosen to engage in, varying from mainly face-to-face to mainly Webex taped have an impact on their performance 
in the course? Two linear regressions were used for this first study. Final exam score was the dependent variable, 
homework score was the independent variable, and attendance behavior was the moderating variable. 

The impact of attendance behavior 

The focus was on the potential moderating effect the behavioral variable might have on the relationship 
between the final exam score and its predictor. The final exam served as dependent variable for this part of the study 
both because of its significant role in the final grade and its strictly controlled environment. The continuous 
variables for this first study were homework score, final exam score, and attendance behavior. Table 1 presents the 
means, standard deviations and Pearson correlations for the continuous variables used in the first part of the study. 



Table 1 

Means, Standard Deviations, and Pearson Correlations for continuous variables 





M 


SD 


1 2 


3 


1 . Homework score 


91.01 


14.40 


.29* 


.15 


2. Final exam 


57.86 


17.48 


- 


.18 


3. Attendance behavior 


2.71 


1.51 




- 



Note: *p < .05; 



The bivariate correlations presented in Table 1 above revealed one significant predictor for the final exam, 
the homework score (r = 0.29) significant at p < 0.05. In a first step the final exam score was then regressed on the 
homework score. The resulted equation accounted for 8% of the variance in the final exam score, F(l,72) = 6.41, p < 
0.05, adjusted = 0.07. In a second step, the interaction between the attendance behavior and the homework score 
was introduced as predictor and mean centered values were used to evaluate a regression analysis (Table 2). 

Table 2 



Summary of Regression Analysis for 2006-2007 dataset 



Predictor 


Beta Weights 

Beta t 


Homework score (A) 


0.30 


2.12** 


Attendance behavior (B) 


0.07 


0.59 


A*B 


0.31 


2.82** 


Model Summary 


R^ = 


0.19, p< 0.01 



Note: N = 74. **p < 0.01 
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In this second step, the interaction term between attendance behavior and homework explained a significant 
increase in final exam score, AR^ = 0.1 1, F(2,70) = 4.78, p < 0.05. Thus, attendance behavior was a significant 
moderator of the relationship between homework score and final exam score (Figure 2). 

Moderation Effect of Attendance Type 



Mostly Taped — ■ — MostlyLi\« Class Mean 




(-1)SD {+1)SD 

Homework Grade % 

Figure 2. Moderation effect of attendance type on homework scores 

The unstandardized simple slope of final exam 1 SD above the mean of attendance behavior was + 0.81, 
and the unstandardized simple slope for final exam 1 SD below the mean of attendance behavior was - 0.08 (see 
Figure 1). The simple slope analysis* indicated that the positive slope of the final exam 1 SD above the mean of 
attendance behavior was statistically significant, t(70) = 3.76, p < 0.05, while the negative slope of the final exam 1 
SD below the mean of attendance behavior was not statistically significant. 

Student attitude toward Webex, the online instructional tool used in the course 

Figure 3 summarizes the mean values for the three measures of students’ attitude toward Webex, the online 
tool uses for the major instructional activities in the course. 




Figure 3. Student attitude toward Webex at the end of the coursework. 
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With a maximum value of nine used for these attitudinal scales, students reported a quite high positive view 
of Webex as online instructional tool for all three measures. More important, they found Webex’s ease of use 
significantly higher than its interactivity and respectively usefulness. That is, paired-samples t test revealed a 
significant difference between mean levels of ease of use (M = 8.21, SD = 1.12) and interactivity (M = 7.23, SD = 
1.37), t(73) = 7.29, p < .05, and respectively between the ease of use and usefulness (M = 7.1, SD = 1.83), t(73) = 
5.47, p < .05. No significant statistical difference was found between students’ ratings of interactivity and usefulness 
of Webex as online instructional tool. 



Discussions and Implications 

The finding of this first study was rather surprising as it suggested that for those students that used mostly 
the online Webex taped lectures, their performance in the final exam was near the mean regardless of their 
homework score. In contrast students that used mostly live lectures performed significantly better in the final exam 
when their homework score was higher. 

Two get more insight on the above finding students in the two extreme of the attendance behavior were 
analyzed with respect of the distribution of their performance in the final exam. The two subgroups considered for 
this analysis were the mostly taped group made of students reporting 90% and up use of Webex taped lectures and 
respectively in the mostly face-to-face group made of students reporting 10% or less use of Webex taped lectures. 

The analysis of the performance of students’ in the two groups showed that: a) students in both groups have 
final exam grades that spread across the entire spectrum of the grading scale (A to F); and b) students in mostly live 
lectures group had more A and less F than the students in the mostly taped lectures group. Figure 4 presents a 
synthesis of these findings. 




□ Mostly Face-to-face (n=17) □ Mostly Taped (n=21) 

Final Exam Grade 



Figure 4. Distribution of grades in the final exam for the two extreme attendance behavior groups. 

These results show that while a large spectrum of students found Webex attractive. Attendance then 
behavior served then as an indicator of students’ confidence in their own ability to learn from online lectures. 
However, some students fully used the benefits of this tool while others where not able to tape in its benefits. The 
fact that the high level of confidence with this tool was not backed up by the level of academic performance can be 
explained by the fact that these students were on-campus students mostly exposed to face-to-face lectures as main 
instructional tool. Since face-to-face lectures require less self-control than online ones, students were less prepared 
to be effective online learners. Other possible explanation emerges from the structure of the course. That is, lectures 
were the main method used to introduce various concepts and their application for the problems analyzed in the 
course. 

Being more prone to disturbance, the taped lectures had then the potential to reduce students’ engagement 
in the learning process, with negative impact on their performance outcome in the final exam. Future research 
should analyze if making students aware of this trap and providing them with effective strategies for online learning 
will close the gap between the performance outcomes of the two attendance groups: mainly face-to-face and mainly 
taped lectures. 
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In recent years, it has been generally acknowledged that there is a need for improved literature 
review skills in educational research. In an effort to address this need, Boote & Beile (2004, 2005) 
proposed a rubric for evaluating the quality of doctoral dissertation literature reviews. This rubric is now 
being used in diverse fields such as nursing (Bowman, 2007), music (Freer & Barker, 2008), and 
information systems (Levy & Ellis, 2006). If literature review skills are important for education research in 
general, then they are also important for Instructional Technology. However, to date, no studies exist that 
examine the quality of dissertation literature reviews in this particular field. Therefore, in an effort to 
further the understanding of the current state of dissertation literature reviews in Instructional Technology, 
a study was conducted replicating Phase 2 of the 2004 Boote & Beile study which examined the quality of 
doctoral dissertation literature reviews in education. 

Practical & Scholarly Significance of Study 

Practically speaking, a well-conducted literature review is central to a scholar’s ability to pose 
pertinent and timely questions within their field. Sophisticated, integrative reviews of literature are critical 
within the field of education as we delve into complex, “hard-to-do” problems (Berliner, 2002). New 
doctoral recipients who have not mastered the skills of reviewing and synthesizing current literature run the 
risk of not understanding the most pressing issues within the field. The consequences can be far reaching, 
especially in a field intertwined with many others such as Instructional Technology. 

From a scholarly standpoint, the research identifies areas of strength in doctoral training in the 
field of Instructional Technology, shedding light on the current state of doctoral dissertation literature 
reviews. The research also serves the purpose of identifying areas needing improvement in the educational 
process of producing scholars who can conduct research that is solidly situated in the current knowledge 
base. This timely research is in keeping with recent initiatives on the doctorate and the process of growing 
new scholars. 

Review of Literature 

There have been several recent major initiatives on the doctorate and the process of “growing” 
new scholars such as the Carnegie Initiative on the Doctorate (Carnegie Foundation for the Advancement 
of Teaching), Graduate Education Initiative (Andrew W. Mellon Foundation), Preparing Future Faculty 
(Association of American Colleges and Universities and the Council of Graduate Schools with funds from 
the Pew Charitable Trust), and The Responsive Ph.D. (Woodrow Wilson National Fellowship Foundation). 
These initiatives have led to an increased awareness of the need for doctoral students to be learning more 
sophisticated research skills (Golde & Walker, 2006; Walker, Golde, Jones, Bueschel, & Hutchings, 2008). 

In an editorial for the Review of Educational Research, LeCompte and colleagues (2003) cautions 
that many new scholars are avoiding investigation at the depth sought by the Review in Educational 
Research because they feel the only “real” research is empirical. The editors go on to say that future 
educational research is weakened because of this, as it leads to research that is not critically integrated with 
prior investigations. They reaffirm that literatures reviews which are “...state-of-the-art literature reviews 
are legitimate and publishable scholarly documents” (LeCompte, et. al., 2003, p. 124) and encourage 
emerging scholars to learn the necessary skills to write such works. 

In her book Making the Implicit Explicit: Creating Performance Expectations for the Dissertation 
(2007), Lovitts offers up insightful rubrics on how to evaluate within a variety of fields of studies, based on 
extensive interviews with seasoned and productive faculty. However she did not examine educational 
dissertations specifically and focused her efforts on the disciplines of biology, physics, electrical and 
computer engineering, mathematics, economics, psychology, sociology, English, history, and philosophy. 
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Consequently, the Boote & Belle (2004, 2005) study remains a singular effort to examine the current state 
of doctoral dissertation literature reviews in education through systematic investigation, yielding 
recommendations based on evidence. 

An adaptation of a rubric first developed by Christopher Hart (1999), the Boote & Belle rubric is 
an ambitious, yet solid synthesis of the many recommendations on improving literature reviews from both 
leading journal editors and academicians. The rubric represents a step forward in the ability evaluate to the 
overall quality of doctoral dissertation literature reviews and in bringing about the changes that have been 
called for. 

The original Boote & Belle (2004) study examined 30 dissertations awarded in the year 2000 from three 
separate Colleges of Education. During Part A of the research study, Boote & Belle conducted a citation 
analysis of the references within the literature review chapters of all 30 dissertations. During Part B, the 
authors examined the doctoral dissertation literature reviews from a more global perspective, focusing on 
the degree of analysis and synthesis within the literature review chapter (traditionally chapter 2). To do 
this, they developed a 12-item rubric based on Hart’s (1999) work. They then selected four dissertations 
from each of the three Colleges of Education to examine in greater detail for a total of 12 dissertations. 
They found that the mean scores for the items ranged from a low of 1.08 (SD=.29) on item A, “justified 
criteria for inclusion and exclusion from review,” to a high of 2.33 on three separate items and across all 
twelve scored literature reviews, the mean statistic was 2.09 (SD =.50). 

In an effort to further the understanding of literature reviews within educational research in 
general and specifically within the field of Instructional Technology, we sought answers to the following 
questions: 

1. What is the overall mean and SD for dissertation literature reviews in Instructional Technology? 

2. Do individual criterion means differ significantly from the Boote & Beile results? 

Method 

Selection 

A purposeful sample (Patton, 2002) of dissertations was collected from top programs in 
Instructional Technology. These programs are not rated by any commonly agreed upon source. However, 
there is an informal ranking of the top five that fluctuates with the changing nature of the various programs. 
Most professionals in the field could easily name the top 10 research oriented programs with little 
disagreement and only some discussion as to how they should be ranked. Prom this informal ranking of top 
programs, five of the top Instructional Technology programs in the United States were selected. 

Department heads and senior faculty were asked by email to make recommendations for their three top 
dissertations from their program during the academic years 2002-2007. No mention was made that the 
study would only compare the reviews of literature. This yielded a total of 15 dissertations. 

Design and Procedure 

Digital Dissertation Abstracts and institutional library catalogs were searched to obtain the 
dissertations. Two trained evaluators conducted a blind review of the literature review sections. They 
individually scored each literature review using the 12 criterion identified by Boote & Beile (2005), rating 
the criterion on a scale from one to three, except for criterion H, which had a scale of one to four. The two 
reviewers then met and discussed discrepancies until a consensus in scoring was reaching. As one item is 
scaled on a four and the others on a three, an overall weighted score was calculated for the current study. 
An independent sample f-test was then used to determine if there was a difference in the mean score for 
each of the criterion between the current study and the original Boote & Beile study. 

Results 

Two of the dissertations were discarded from the study as they did not contain a specific literature 
review chapter. One was qualitative in nature and the other was a multiple paper format. The 13 remaining 
dissertations were examined. The mean overall weighted score for the dissertations reviewed in this study 
is 23.00 (SD = 7.24) out of a possible 37. The lowest overall weighted score for this study was 12.98 and 
the highest overall weighted score was 33.44. In regards to the individual criterion, the lowest scoring was 
(A) “Justified inclusion/exclusion,” (M = 1.31, SD = .63). The highest scoring criterion was (H) “Identified 
main methods & advantages/disadvantages,” (M = 2.54, SD = 1.27). There were three criterion which all 
had the same mean score of 2.31, (E) “Acquired/enhanced subject vocabulary,” (M = 2.31, SD = .75), (P) 
“Articulated variables/phenomena relevant to topic,” (M = 2.31, SD = .63), and (L) “Coherent/clear 
structure that supports review,” (M = 2.31, SD = .85). 
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For each of the 12 criterion identified by the Boote & Beile rubric, an independent sample f-test 
was performed comparing the mean scores of the current study with the mean scores in the Boote and Beile 
study. At an alpha level of .05, none of the 12 criterions differed significantly. That being said, three of 
the items yielded virtually the same mean scores for three items. These items were criterion (C) “Placed 
topic/problem in broader literature,” (E), “Acquired/enhanced subject vocabulary,” and (F) “Articulated 
variables/phenomena relevant to topic.” 

For item C, the Boote and Beile mean score was 2.17 (SD =1.03) and the current study had a 
mean score of 2.23 (SD =1.03). The independent samples f-test yielded a score of f(23) = -.17, p = .87. For 
item E, the Boote and Beile mean score was 2.33 (SD =.49) and the current study had a mean score of 2.31 
(SD =.75). The independent samples f-test yielded a score of f(23) = .10, p = .92. For item F, the Boote and 
Beile mean score was 2.33 (SD =.49) and the current study had a mean score of 2.31 (SD =.63). The 
independent samples f-test yielded a score of f(23) = . 1 1 , p = .91. 

While not being significant but of interest nonetheless, the items that had the least agreement in 
mean score were (G) “Synthesized/gained new perspective on literature,” (H) “Identified main methods & 
advantages/disadvantages,” and (1) “Related ideas/theories to research methodology.” For item G, the 
Boote and Beile mean score was 1.42 (SD =.67) and the current study had a mean score of 2.00 (SD 
= 1.00). The independent samples f-test yielded a score of f(23) = -1.70, p = .10. For item H, the Boote and 
Beile mean score was 1.92 (SD =.79) and the current study had a mean score of 2.54 (SD =1.26). The 
independent samples f-test yielded a score of f(23) = -1.15, p = .16. For item 1, the Boote and Beile mean 
score was 2.17 (SD =.84) and the current study had a mean score of 1.69 (SD = .75). The independent 
samples f-test yielded a score of f(23) = 1.50, p = .15. A possible reason for these scores is discussed below. 

Discussion 

Overall, our findings support Boote & Beile’ s original assumption that there is room for 
improvement of doctor dissertation literature reviews in education. Even among the best examples of 
doctoral dissertations from the top programs in Instructional Technology, there was still significant room 
for improvement as the average weighted score for the top dissertations was 23.00 (SD = 7.24). While it is 
interesting to note that item (H) “Identified main methods & advantages/disadvantages,” (M = 2.54, SD = 

1 .27) had the highest mean score, caution should be exercised in that this item was assessed on a scale of 
four, unlike the other 1 1 items which were assessed on a scale of three. Item (A) “Justified 
inclusion/exclusion,” (M = 1.31, SD = .63) had the lowest mean score which is surprising considering the 
importance of delineating the parameters of article selection for a literature review. 

The three criterion [(C) “Placed topic/problem in broader literature,” (E), “Acquired/enhanced 
subject vocabulary,” and (F) “Articulated variables/phenomena relevant to topic.”] with the least deviance 
between the current study and the Boote and Beile study are typically taught in general education research 
methods textbooks. Most students will learn these skills in their basic educational research methods classes. 
Therefore, we would expect that the current study would not differ significantly from the Boote and Beile 
study as these skills are generally criterion that doctoral candidates are familiar with. Our results for these 
three items demonstrate the reliability of the rubric in measuring these particular outcomes. 

The current study’s higher mean scores on (G) “Synthesized/gained new perspective on literature” 
and (H) “Identified main methods & advantages/disadvantages,” may be accounted for by the fact we 
examined the best dissertations literature reviews in our field. Presumably, departmental representatives 
selected these dissertations because they examined problems in new ways and offered a new perspective or 
methods. The lower deviance score on criterion and (1) “Related ideas/theories to research methodology” 
may be attributed to a systemic issue within the field in that we draw from a broad base of related 
disciplines 

This study is not without its weaknesses. The purposeful sampling of a small number of 
dissertations selected for their extreme characteristics of being judged to be some of the best dissertations. 
Also, the current study did not examine the whole dissertation. This is problematic in that the criterion 
identified by Boote & Beile may be contained within other chapters of the dissertation and not solely 
situation within chapter 2. Additionally, inherent weaknesses in the rubric became apparent in it’s 
application during this study. All of the items are not on the same scale, i.e., criterion H was scale on a four 
instead of a three like the rest of the criterion. Additionally, the conceptual levels between the various cells 
in the rubric were unequal; at times the threshold to get a score of three instead of two was not the same 
between items. This resulted in some difficulty in the actual scoring of each dissertation as it was 
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frequently a “judgment call” for each reviewer whether the dissertation met a particular cells requirement 
threshold to earn the next highest score or not. 

Recommendations for Future Research. In a broader perspective, future research should focus on a 
randomized replication study with a larger n; it should also examine the entire dissertation for evidence of 
literature review skills, not just chapter two. Also, Boote and Beile contend the skills needed to craft a well- 
written, integrative literature review are not found in textbooks. A systematic review of the top textbooks 
used in education research methods classes is needed to determine if this is in fact true. 

As the rubric is being used in diverse fields, further studies should be conducted in the cross- 
discipline use of the rubric; can it or should it be applied to other fields such rather that just the field of 
education? Also, a qualitative study of the authors of the top-rated dissertations should be conducted to 
identify where the learned the necessary skills to produce a high quality literature review for their 
dissertation. Work also needs to be done on the rubric itself to provide for consistent scaling of each item 
and conceptual equality between the cells in the rubric. 

In a more focused application, future scholarly research in instructional technology regarding how 
to improve doctoral dissertation literature reviews should encompass three areas: identifying how literature 
review skills are taught at the doctoral level, improving how literature review skills are taught at the 
doctoral level, and the adaptation of the Boote & Beile rubric for specific use within Instructional 
Technology. Critical voices in the past 10 years have called for more rigor in the research emerging from 
education. Improving the quality of our literature reviews provides a solid foundation for that increased 
rigor. 
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Table 1 



Boote & Beile Scores Compared to Current Study 



Criterion 


Study 


Mean 


SD 


Sig. 


A. Justified inclusion/exclusion 


Boote & Beile 


1.08 


.29 


.27 




Current Study 


1.31 


.63 




B. Distinguished what has been done 


Boote & Beile 


2.17 


.72 


.42 


from what needs to be done 


Current Study 


1.92 


.76 




C. Placed topic/problem in broader 


Boote & Beile 


2.17 


1.03 


.87 


literature 


Current Study 


2.23 


.83 




D. Place topic/problem in historical 


Boote & Beile 


2.33 


.78 


.38 


context of field 


Current Study 


2.08 


.64 




E. Acquired/enhanced subject 


Boote & Beile 


2.33 


.49 


.92 


vocabulary 


Current Study 


2.31 


.75 




F. Articulated variables/phenomena 


Boote & Beile 


2.33 


.49 


.91 


relevant to topic 


Current Study 


2.31 


.63 




G. Synthesized/gained new perspective 


Boote & Beile 


1.42 


.67 


.10 


on literature 


Current Study 


2.00 


1.00 




H. Identified main methods & 


Boote & Beile 


1.92 


.79 


.16 


advantages/disadvantages 


Current Study 


2.54 


1.26 




I. Related ideas/theories to research 


Boote & Beile 


2.17 


.84 


.15 


methodology 


Current Study 


1.69 


.75 




J. Rationalized practical significance 


Boote & Beile 


2.17 


.58 


.68 




Current Study 


2.08 


.49 




K. Rationalized scholarly significance 


Boote & Beile 


1.92 


.79 


.36 




Current Study 


1.62 


.77 




L. Coherent/clear structure that supports 


Boote & Beile 


2.08 


.67 


.48 


review 


Current Study 


2.31 


.85 
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Abstract 

National emphasis on standards requires that pre-service teachers learn to integrate technology into 
classroom practice. Digital storytelling is a culturally authentic form of representation that integrates writing, 
communication, and media literacy with technology skills. Pre-service teachers develop planning skills, personal 
voice and an understanding of narrative structure as they construct digital stories. Using both quantitative and 
qualitative methods, this pilot study investigates pre-service teachers’ perceptions of the value of adding digital 
storytelling to a writing intensive children’s literature class. 



Introduction 

Current emphasis on technology integration of state mandated and ISTE/NETS standards into pre-service 
courses requires educational institutions to prepare pre-service teachers to be effective users of technology. The task 
is two-fold; providing pre-service teachers with technology skills and also developing a valuing of technology and 
an understanding of how to utilize technology as a tool for teaching and learning. Success in technology integration 
is best measured by the change in students’ perceptions of themselves as technology users, their comfort level in 
using technology, their ability to design lesson plans with technology, and students’ valuing of technology as an 
integrated component of teaching and learning (Collier & Veres, 2006, Hughes, 2000). 

Pre-service teachers are preparing to teach in a contemporary culture of technology in a society that 
demands that students be able communicate effectively in both writing and speech. Pre-service teachers must learn 
to communicate effectively themselves if they are to teach the young. They must also learn to integrate technology 
into the curriculum as a tool for learning and communication. According to Bruner (1990, 1996), education is a 
major embodiment of our culture. If pedagogy is to empower students to learn, it must be able to transmit the current 
cultural tool kit. Digital storytelling is a contemporary approach to traditional oral storytelling that utilizes 
multimedia and telecommunication tools to engage students in authentic learning experiences that provide real 
world relevance and personal meaning to learners (Bruner 1996, Brown, Collins, & Duguid, 1989, Kearney & 
Schuck, 2006, Emihovich & Lima (1995), Lambert, 2003). 



Purpose of the study 

The purpose of this study is to investigate pre-service teachers’ perceptions of the value of adding digital 
storytelling to a writing intensive children’s literature class. The research seeks to determine the value added 
component of moving from writing in a text-based format to the digital environment that adds visual, symbolic, and 
auditory components to the writing process and leads students through an iterative, reflective cycle to develop an 
artifact that can be shared with others and also reflected on and evaluated by the author. The study examines digital 
storytelling as a strategy for teaching the process writing cycle: planning, writing, reflecting, self-evaluating, and 
revising; increasing personal voice in writing; enhancing an understanding of narrative structure; and developing 
technology skills. The study also looks at pre-service teachers’ beliefs about the value of digital technology as a 
teaching and learning method and their future plans to incorporate digital storytelling into classroom practice. 

Theoretical framework 

Education is a major embodiment of our culture (Bruner, 1996). The culture of the twenty-first century is a 
digital culture. “School children today are bombarded with images, video, sound and media from television, radio, 
the World Wide Web and even their own cell phones” (Hofer & Swan, 2006). New digital technologies are 
transforming how we teach, learn, and communicate both in and out of the classroom (Weis, Benmayor, O'Leary, & 
Eynon, 2002). Ideally, these new digital technologies hold the potential to transform classrooms from spaces of 
delivery to spaces of active inquiry where students are empowered to become researchers, storytellers, scientists, 
and historians (Hofer & Swan, 2006; Weis, et ak, 2002). 

This change can only take place when pre-service teachers learn to effectively utilize digital 
technologies as tools for thinking, learning, and sharing ideas. Digital storytelling integrates oral 
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storytelling with modern technology and utilizes multimedia and telecommunication tools to engage 
students in authentic learning experiences that provide real world relevance and personal value to the 
learner within a situated context (Bruner 1996, Brown, Collins, & Duguid, 1989, Kearney & Schuck, 

2006, Emihovich & Lima, 1995, Lathem, 2005, Lambert, 2006). Stories serve as avenues to personal 
experience, and they are the way that human beings make sense of the world and create a personal reality 
(Bruner, 1987, 1991, Schank, 1990, Davis & Waggert, 2006). It is through telling stories that students 
develop voice and a sense of personal identity (Schank, 1990; Barrett, 2005). 

Bandura, 1989, 2001, 2006, states that being an agent is to influence intentionally one’s functioning and 
life circumstances. Agency includes the ability to make intentional choices and to develop and carry out a plan. 
Digital storytelling is a process that blends photographs, drawings, video, music, narrative, and voice to create a 
multimedia story that can be biographical, fictional or documentary (Burk, 1997, Lambert, 2006). Designing a 
digital story requires students to write a story script, select music and images or video to elaborate the text, and 
record a voiceover. Students must make choices and engage in intentional planning. Students evaluate their stories 
in a reflective process that leads to an iterative progression of self-evaluation, editing and revision. Collier & Veres 
(2006) state that digital storytelling results in an authentic product that engages pre-service teachers in using their 
knowledge and skills simultaneously. 

Language is the primary way in which human beings create meaning. Language mediates both 
communication, which enables thinking with others, and also the inner speech through which individual thinking is 
brought under conscious control (Vygotsky, 1986, Wells, 2007). Semiotic mediation is characterized by the use of 
sign systems which serve as abstract tools to change the nature of human mental functioning. Vygotsky linked 
learning to thinking. “Properly organized learning results in mental development” (Vygotsky, 1978). Vygotsky 
stressed the centrality of the active construction of knowledge and the impact of culture on individual development 
theorizing that understanding is social and cultural in origin (Cole & Wertsch, 1996). 

Jonassen (1996) repurposes the computer as a cognitive tool, a Mindtool or transparent meditational artifact 
for the analysis and construction of knowledge. “Technologies should not support learning by attempting to instruct 
learners, but rather should be used as knowledge construction tools that students learn with, not from. . . .Learners 
should function as designers, and computers should function as Mindtools for interpreting and organizing their 
personal knowledge” (Jonassen, Carr & Yueh, 1998). Mindtools are computer applications that provide learners 
with a means to represent what they know and engage students in critical thinking about the content they are 
studying. “Mindtools guide learners to organize and represent what they know and engage students in reflection and 
deeper analytical ability representative of constructivism thinking” (Jonassen et ah, 1998). 

Digital storytelling extends the oral tradition and transforms storytelling into a digital artifact that can be 
stored, shared with others, and reflected upon by the author. Designing a digital story engages students in using a 
variety of technology applications to organize, edit, and assemble a digital story. When computers function as 
intellectual partners for the construction of knowledge, they share the cognitive burden of carrying out tasks 
(Saloman, Perkins, & Globerson, 1991). Digital storytelling combines content learning and discipline specific 
processes such as narrative structure and the reflective writing cycle with 2L‘ century technology skills in student- 
centered engaging ways (Hofer & Swan, 2006). 

Study participants and the digital storytelling process 

Participants in this study include pre-service teachers enrolled in two sections of Reading 302, Teaching 
Reading through Children’s Literature at Texas A&M University (N = 71 females, 1 male); (African American (1), 
Caucasian (65), Hispanic (6), Asian (0). Reading 302 is a one-semester introductory children’s literature course 
designed to prepare teachers in Early Childhood Education to teach critical reading, language arts, and children’s 
literature. The course is project-based and emphasizes both competent writing skills and effective methods to teach 
writing to early childhood - 4* grade students. The digital storytelling project takes place over eight class periods 
with the final two periods used to view the digital artifacts. The digital storytelling project is the culminating project 
of the semester and draws on knowledge from prior projects on picture story books and oral storytelling methods, 
literary elements and narrative structure, and the reading/writing workshop process. Two class periods are devoted 
to discussion of the elements of digital storytelling, a demonstration of software required to record narrative and 
construct a digital video, and viewing examples of digital stories. Students are also provided with a one page 
handout on how to construct a digital story. A story telling circle is a meaningful part of the digital storytelling 
process to allow participants to share thoughts and receive feedback. Due to time constraints, this project used a 
virtual storytelling circle on the class website. Students posted their story ideas to the class website. Using peer 
review techniques, each student replied to two postings with suggestions and comments about the story topic as a 
guide to the author. Next pre-service teachers wrote a 200-500 word personal narrative about a meaningful 
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experience. This personal narrative becomes the basis for a digital story. Pre-service teachers are also asked to 
locate appropriate images, video clips, and music to illustrate the story. Recording the personal narrative using 
Audacity®, a free software program for capturing and editing audio, is the first step in the construction process and 
establishes the framework for story. The recorded narrative is then combined with images, video, and music to 
create a digital story using Windows Movie Maker®. The reading/writing process, which includes pre-writing, 
reflection, self-evaluation, revision, and conferencing with faculty, is employed throughout this project. Students 
spend four class periods in the computer lab, with two extended time periods, editing the written narrative, recording 
the audio track and putting the digital story together. In the computer lab, Reading 302 faculty and the researcher 
helped students with the editing process and support the development of technology skills. 

Research methodology 

This research employs a mixed methods approach. A twenty-five question, five point Likert scale attitude 
survey was given at the beginning of the project and at the final class of the semester to assess changes in students’ 
attitudes toward agency and self-regulation, digital storytelling as a teaching and learning strategy, the role of story 
in teaching, the value of the writing process, and beliefs about digital technology. 

The researcher took the role of participant/observer during the eight class periods of the digital storytelling 
project which allowed for interaction with all participants during the digital story construction process and for 
viewing all completed artifacts. Voluntary interviews, conducted after the completion of the digital storytelling 
project, employed a semi-structured script. Interviews were recorded and transcribed. The constant-comparative 
method of analysis was used to code, categorize and analyze the interview data. 

Findings: quantitative 

A paired sample t-test was conducted to evaluate the Likert scale survey pre-post data. Of 72 students in the 
two sections of Reading 302, 35 participated in both the pre and post tests. The paired sample t-test evaluated 
whether there was a change in students’ attitudes toward self-regulation, the writing process, computer use in the 
classroom, the value of stories, and the value of digital storytelling as a teaching strategy as a result of the 
implementation of the digital storytelling project. Results indicated an overall statistically significant increase in the 
mean difference in students’ attitudes. The post-test mean (M= 94, SD =7.99) was significantly greater than the pre- 
test mean (M= 90.20, SD = 8.87), t (34) = 3.27, p<.01. The standardized effect size index, d was .55. The 95% 
confidence interval for the mean difference between the two ratings was 6.16 to 1.44. The homogenous nature of the 
pre-service student population, which is composed primarily of Caucasian girls in their early 20’ s, limits the choice 
for analysis for the Likert scale data as there are no significant within group or between group factors. 

Findings: qualitative 

Seven female pre-service teachers volunteered to participate in individual interviews. Interview questions 
focused on the difference that students experienced when they moved from the familiar text-based writing format 
into a digital environment that adds visual, symbolic, and auditory components to the writing process and results in 
an artifact that can be shared with others and also reflected on and evaluated by the author. Specific areas of interest 
included aspects of agency such as the planning process and motivation, development of personal voice and image 
as language, the definition of story and understanding narrative structure, feelings about digital technology and 
future plans for using digital storytelling in the classroom (Glaser & Strauss, 1967; Merriam, 1998). Findings are 
organized according to these areas. 

Agency 

To be an agent is to intentionally make things happen by one’s actions (Bandua, 2001). Agency includes a 
commitment to carry out a plan, forethought, outcome expectations based on realistic goals that motivate and guide 
personal actions, self-reactiveness, the capacity to self-regulate and motivate, and self-reflectiveness, the capacity to 
evaluate one’s actions. Pre-service teachers stated that they employed personal planning strategies, some structured, 
others open-ended; however, all indicated that the planning process involved forethought, reflection, and self- 
assessment. Several students indicated that linking narrative and images was integral to writing in the digital 
environment. One student used a storyboard to structure her experience and then wrote her story; another wrote the 
story first and then drew the images required to illustrate her text, a third located photographs first and then wrote 
her story. Two relied on a free writing and revision process that is directly related to prior experience in the text 
based environment. 
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“It’s a process of putting things together. I planned out the images need to tell the story first; then wrote the words 
each picture would require to tell the story. I used a story board format and linked the images to the text to achieve 
the message I wanted to tell. The images carried much of the message. Images and text are integrally linked to 
convey the meaning of the story. ” 

“I wrote and recorded before I did the pictures; then I went through and found the sentences that I wanted to have 
pictures for. I actually think the pictures that I drew were more personal than photographs because I could actually 
draw what I felt rather than this thing of clip art of a girl crying or somebody playing the piano. ” 

“I just write and the ideas just kind of flow. I went home and started writing my rough draft the day we started it. It 
wasn’t like I had to do it. It was fun to do. I can’t draw, so I thought about what I had the most pictures of. I tried to 
think about what was significant; then I just kind of started writing stuff. That’s how I write. I don ’t like to outline or 
anything. I just write and get this paragraph and then I put in another paragraph. ” 

“I just wrote down the thoughts as they were coming through my mind. I didn ’t use a plan. I had four drafts. But 
that helped... doing the drafting process and getting feedback. You put your ideas, your thoughts down without 
paying attention to whether it makes sense; just your thoughts being flowed into a pencil. ” 

Motivation 

Pre-service teachers were particularly motivated by the knowledge that they were designing a digital 
artifact that could be used in a future classroom. “We actually had a product to show for our work. It was more than 
just a paper you write, and hand in. It’s like you have a reason... like it is authentic... creating for a purpose rather 
than merely being graded. ” One wrote about her first day in a new school. She said that showing The New Girl 
would be a good way to engage children in talking about sensitivity to the feelings of others. 

“/ wanted to do something that was different, creative and useful I planned the story to be a product that I 
could use in a 4* grade class where I hope to teach. I chose a message that elementary students could 
understand... a story about someone who feels left out, a new girl. 

Another wrote about an upsetting personal experience, playing the piano for the first time in a recital. This 
pre-service teacher plans to show The Recital to introduce herself to a new class. She explained that the story reveals 
her humanness, and she wants students to know that expressing personal emotions is acceptable. 

“/ guess my story would convey that you can talk about anything and that it’s alright to show emotion. Like in my 
story, I cry. It would convey that it’s OK to talk about real things instead of just. ..One day I found a key. It makes 
them feel more comfortable with their own emotions and feelings. ” 

Personal Voice and Image as Language 

Voice is the writer coming through the words, the sense that a real person is speaking to us and cares about 
the message. It is that individual something, different from the mark of all others writers, that we call voice 
(Northwest Regional Education Lab, 2001). While language, spoken and written, is the primary way in which 
human beings create meaning, signs and images also serve as abstract tools for thinking. (Vygotsky, 1986, Wells, 
2007). 

Hispanic pre-service students indicated that digital storytelling made them feel more comfortable about 
expressing personal voice, sharing a story with others. Not only could the narrative be recorded in private, but they 
could also use images to express their ideas. A future ESL teacher commented that the technology component of 
digital storytelling made her feel more comfortable telling her story because she has an accent when she speaks 
English and gets nervous when she has to speak in front of a big crowd. She was inspired by the idea of creating a 
project that she could use with ESL students because a digital story will help ESL students make a connection 
between an image, the word in English and the word in Spanish both visually and auditory. This student told a story 
about being bitten by bees as a child. Why Are Mom ’s Always Right! To illustrate her story, she designed collages 
that included photo cutouts of family members and clip art pictures. She utilized the background color along with 
lines and shapes cut from construction paper to convey the emotional context of each frame. Key words used in the 
story such as abeja (bee), panal (beehive) and fin (end) are written in both English and Spanish. Each panel of her 
story is designed as a teaching tool that can be easily interpreted by a non-English speaking viewer. Her screams of 
pain from the bee stings are understandable in any language! 
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“The images were crucial for the story because the people I wanted to target are English Language Learners so the 
way that they are able to understand the plot of the story if they happen to not know a word is by looking at the 
picture. I also added words in both languages in order to enrich their vocabulary and with the thought in mind that I 
can use that story to teach a science lesson and a language lesson. I wrote in small groups of sentences and 
sketched out a small drawing of the image. I used lines to create energy in the drawings and mood was created with 
color. I used both English and Spanish in the story, so that second language users could relate key images to the 
Spanish word. ” 

Another ESL pre-service teacher, who is originally from El Salvador, explained that the digital format 
allowed her to share her culture with other students. She commented that in a large university class, students don’t 
have the opportunity to get to know one another. Her digital story. Going Home, allowed this student to speak in her 
own voice and through images, to share her life story with classmates and establish her individual identity within a 
larger dominant group. 

It felt good to share what I had written with the rest of the class. Being exposed to a video format, they can see a 
diverse culture. I wanted to relate the experience of going home... what it was like for me to go back and relive my 
childhood memories. It was that experience that I wanted to express. . .just going back to those childhood memories. 
We learn through social interaction in a cultural context. 

Definition of story and understanding narrative structure 

“A narrative is an account of events occurring over time” (Bruner, 1991). A story has a specific syntactic 
shape or structure: beginning-middle-end or situation-transformation-situation and must contains three basic 
elements: a situation that involves a conflict or predicament; an animate protagonist who engages in the situation for 
a purpose; and a sequence with implied causality during which that predicament is resolved (Scholes, 1981). Prior to 
the digital storytelling project, pre-service teachers in Reading 302 participated in a literary element project that was 
designed to teach story structure, and the basic element of story; setting, characterization, theme, plot, conflict, and 
author’s style. The researcher wanted to know if the digital storytelling project reinforced and extended students’ 
understanding of story structure as a result of taking these concepts into a very visible format, a digital story. 

Students who were interviewed had some form of definition for story, but only one knew the meaning of the term 
plot. Definitions for story did not include the concept of conflict and resolution. Although students were able to 
identify story elements in coursework, most were not able to connect the elements of narrative into a conceptual or 
“mental model” of a story and were therefore unable to meaningfully integrate them into their own stories. 

“Story is a narration of something. It can be an event, a dream. ” 

“It’s just an event. A story has a plot. ” 

“Someone’s experiences or imagination. A story is just something that someone thinks is important. ” 

“Stories are entertainment. They have characters, a point of view. An event has to take place. ’’ 

Technology and future classroom use 

Analysis of interview data revealed that pre-service teachers are highly motivated by the process of 
designing a digital product that can serve as a useful artifact in a future class. Students indicated that prior 
technology training occurred primarily in high school; not in the university environment. Most were familiar with 
word processing, spread sheet and presentation programs, but only a few had ever recorded audio or used a video 
editing program. As a result of the project, students gained confidence in digital technology use as well as an 
increased valuing for how digital storytelling can be integrated into curriculum. 

I just mainly work with Word and PowerPoint, and those were really just minimal... typing and a few equations. 

I had never made a video. When I was little, we used to record it on a tape recorder, but never into a microphone 
much less into a computer. I had scanned like two pictures years ago, but it’s not hard 

I didn’t know how to do anything, except to type. But like going to a web-site and downloading things, I have never 
done that. I have used a scanner before. I do have one at home, but I have not used it to scan photos. 
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The most significant finding of this pilot project is the perceived benefit of digital storytelling for pre-service 
teachers, not only for their own education, but also for their future classrooms. In interviews, pre-service teachers 
revealed that they enjoyed learning new digital technology skills and felt empowered by the process of putting 
together a digital story and sharing the videos with classmates and indicated that that they would use technology in 
future classrooms. 

During the last two periods of Reading 302, classes viewed the digital stories. Responses from pre-service 
teachers reflect a new confidence in digital technology skills and a positive change in attitude towards using digital 
technology in a future classroom. When asked how they felt when their digital stories were shown in class, the 
resounding response was... I was proud. Two comments illustrate pre-service teachers’ enthusiasm for digital 
storytelling and changed attitudes towards using digital storytelling in a writing intensive class. 

“Digital storytelling made writing more interesting. When you write in a regular text-based assignment, you don ’t 
see the outcome. You don ’t see it in picture format. I was able to see a vivid image of what I wrote. That made it 
more interesting and appealing to me. ” 

“/ was really proud of myself when we were all done, and we showed it in class. You don ’t think you could create 
something when you have never done it before. I didn’t know that digital storytelling existed. I have never heard of it 
before. So that’s pretty cool. When you get into a digital story, it is really fun to tell your own story. ” 

Lessons learned 

As a result of this pilot project, several changes will be made in the next iteration of this study. Watching 
the digital stories in class revealed that many students do not understand the difference between a story, which has a 
plot, conflict and resolution, and a documentary that is simply series of images rather like a slide show. Reading 302 
is a writing intensive course and the focus of the digital storytelling project must be on writing first; then on the 
skills to made the narrative into a digital story. Students should write and edit their digital stories before any 
discussion about technology takes place or any examples of digital stories are shown. Otherwise students tend to 
focus on photographs instead of story. The project must be based on writing; not on a series of images pulled from a 
scrapbook or last summer’s vacation photos. 

The assignment for this pilot project asked students to write about a meaningful experience. Many students 
in their early 20s don’t seem to have a clear idea of what constitutes a meaningful experience or perhaps the 
expectations of the researcher and the faculty was just different from students’ ideas. Carter (1993) suggests using a 
focused writing prompt to write a personal narrative that represents“well-remembered event” in students’ lives. We 
suggest using a story example to discuss what constitutes a well-remembered event. 

The virtual storytelling circle will be used again, but students will be required to post a draft of a completed 
story instead of a story idea. Students commented that feedback from other pre-service teachers was shallow and did 
not help to narrow ideas. Appropriate feedback responses will be modeled by faculty before a virtual storytelling 
circle is used again. 

As mentioned previously, students study the elements of narrative, but they don’t necessarily integrate the 
terms into an understanding of story structure. According to Bruner, there are many conventions for expressing 
narrative sequence or plot; however, what underlies all forms for representing narrative is a mental model whose 
defining property is its unique pattern of events over time (Bruner, 1991). Students will continue to study the 
elements of narrative in the literary elements project, but the learning process will be scaffolded with a concrete 
model, VPS (Visual Portrait of a Story). Students will be required to plot the action of the story using the model. 
(Dillingham, 2005). 

Pre-service teachers did use a variety of planning strategies to design digital stories; however, many did not 
follow the reading/writing process that was covered in a prior project. After a review of the digital artifacts, the 
researcher concluded that the planning process should include the use of a storyboard that requires students to link 
the narrative segments of the story to images that extend meaning. 

Conclusion 

Mergendollar (1996) cautions, “While technological tools can spur pedagogical changes, the utility of such 
changes must be measured ultimately by their impact on student learning.” Success in technology integration is best 
measured by the change in students’ perceptions of themselves as technology users, their comfort level in using 
technology, their ability to design lesson plans with technology, and students’ valuing of technology as an integrated 
component of teaching and learning (Collier & Veres, 2006, Hughes, 2000). A second iteration of this study will 
take a more structured approach to the digital storytelling process; however, results of this pilot study indicate that 
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digital storytelling is an effective strategy to support the development of agency, voice, and technology skills in pre- 
service teacher and provide pre-service teachers with a meaningful approach to technology integration that they can 
carry into the K-12 classroom. 
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Introduction 

Background 

Nowadays, online learning or e-Learning is developing a unprecedented prevalence. Growing with the 
number of the online courses that have been offered is the number of online instructions that involve team projects. 
The increasing reliance on technology opens more possibilities for teams to meet virtually (Griffith & Neale, 2001). 
Especially in recent years, the increase of virtual teams used in both academics and business are explosive (Fiol & 
O’Connor, 2005; Piccoli, Powell, & Ives, 2004). Virtual teams overcome the limitations of time, space and 
organizational boundaries that traditional teams faced. But they also face obstacles that traditional face-to-face teams 
do not have, such as the limited technical abilities to work on virtual teams and lack of collaboration skills in a 
distributed environment. 

Purpose of Research Study 

The purpose of this research study is to explore the issues and best practices related to virtual teamwork 
and propose suggestions for improvement on the effectiveness of virtual teamwork. 

Research Questions 

The research study is seeking to answer the following three questions: 

1 . What are the issues and challenges faced by students working in virtual teams? 

2. What are the best practices and strategies for student’s learning through virtual teams? 

3. What are the best practices and strategies for the instructor to facilitate student’s virtual teamwork? 

Review of Literature 

According to Lipnack and Stamps (2000), a virtual team is defined as a group of people who work 
interdependently with a shared purpose across space, time and organizational boundaries using technology. Fiol and 
O’Connor (2005, p20) further defined virtualness “as the extent of face to face contact among team members.” They 
also differentiated teams into three categories: pure virtual, hybrid and face-to-face teams. For the purpose of this 
study, we consider both pure virtual and hybrid as virtual teams. Given virtual teams are a relatively recent 
phenomena, there are major discrepancies between the available resources to increase the effectiveness of teachers 
who teach online and the resource available to instructors who teach face-to-face (Napier & Waters, 2001). Similar 
discrepancies also exist between students who work in virtual teams and those who work in traditional face-to-face 
teams. 

Dede (1996) claimed that collaborative online learning can help get acquisitions of complex and higher 
level concepts and skills that have been a weakness of traditional non-interactive distance education. Slaven (1990) 
recommended using collaborative learning to develop an online community. Collaborative learning should include 
sharing learning task, combining expertise, knowledge, and skills to improve the quality of learning process. Virtual 
teams are just like a vehicle to achieve collaborative learning in distance education. 

The review of literature revealed a number problems and issues related to virtual teamwork in distance 
education. For example. Shin (2005) states that either limited or no face-to-face interaction, make it difficult for 
team members to receive guidance or instructions from supervisors or team leaders. Moreover, it gives fewer 
opportunities to clarity role assignments. Therefore, virtual team members are more likely to get into ambiguity. 

Another problem may be generated by culture difference. People from different cultures and different 
backgrounds vary in terms of their communication styles. They also differ in personality, which may cause 
communication conflicts in virtual teamwork. Hofstede (1980) maintains that people from individualistic cultures 
emphasize on the need, values and goals of the individual than those of the group. Virtual teamwork implies that 
people from different background work together and they might have different perspectives to solve the problems, 
so how do team members get a consensus? Alpay (1998) demonstrates that if team members fail to at least negotiate 
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some common ground, their problem-solving process will close down. However, Kirschner and Bruggen determined 
“in education, such cognitive conflicting is seen a stimulus for knowledge negotiation and construction. 
Argumentation among participants in a collaborative learning situation is therefore stimulated. Yet, the balance 
between maintaining common ground is a delicate thing” (Kirschner and Bruggen , 2004, pi 36). 

In sound virtual team environments, assignments and projects can be worked out through self-direction and 
collaboration. However, putting students into teams does not guarantee collaboration. Kerr & Braun (1983) 
identified the “free-rider” in virtual teams. The free- rider exists when some team members feel that the team is 
doing enough and they do not need to contribute. In addition, “sucker effect” arises when the team members exert 
less effort as they become awared of the peer members’ increased free-riding. 

Another problem addressed by Jansen, Van Laeken & Slot (2003) is that virtual team members tend to 
concentrate on project work and neglect their learning tasks. He stated that learners pay more attention on how to 
finish their assignment and neglect tasks such as reflecting on process feedback from others. 

Successful virtual teams have certain characteristics and features that we will explain and to give some 
guidance on how to effectively work in virtual teams. Based on a case study of collaboration in virtual team, Lewis 
(1998) identifies key factors for ensuring effective virtual teamwork, including; a) social interaction to build trust 
and maintain motivation; b) sharing tacit knowledge to build and maintain common understanding, again building 
trust and reducing feelings of exclusion; c) leadership styles that change according to different situations; and d) 
explicit member roles and responsibilities. Lewis (1998) also states that trust is a crucial element in developing 
effectiveness in virtual teamwork. He suggests in order to creating a good start for a virtual team, members need to 
trust each other and they need to meet in person at first. Alternately, Hackman (1990) described the effectiveness in 
virtual teams in three criteria: deliver on objectives, meet the psychological needs of its members and work together 
in the future. 

Lack of feedback or delays in receiving feedback from instructors may cause students frustration or anxiety 
which affects the team effectiveness (Hara & Kling, 2000). Team members and team leaders also need to work 
together in sharing resources, helping each others solve problems, providing encouragement, and responding to 
requests promptly. Establishing effective communication is essential for learning in virtual teams. There are some 
recommendations for effective online communications; a) team leaders should specify clear role assignments and 
requirements before starting the project; b) ask questions when messages are unclear; and c) encourage open 
communication which means we can incorporate informal communication, humor and honest communication. 

Methodology 

Research Settings/Context 

This qualitative research study, a collaborative work of three researchers, focuses on individual experiences 
in the hope to identify the common issues and best practices of virtual teamwork in online learning environment. 
This study is also seeking any suggestions and personal opinions toward successful virtual teamwork from the 
participants. The study was conducted within the College of Education at a south-eastern state university. Most of 
the participants are graduate students in the Instructional Technology program. They all have some experiences of 
taking online courses and working in virtual teams. 

Sampling Strategies & Sample Size 

In total, nine participants took part in this study. Criterion sampling strategies were employed for this study. 
Participants with virtual teamwork experience were recruited among the graduate students from the College of 
Education at the university. 

Data Collection 

Interview was the method used in this study for data collection. Each interview took 25-35 minutes 
depending on the participant’s involvement. Offices, conference rooms and student labs were used as the interview 
site. Digital voice recorders and notebooks were used to assist data collection during the interviews. Before the 
interviews, informed consents were collected from the interviewees. Right after each interview, the researchers 
worked out the transcriptions with Microsoft Word and added notes for improvements on next interview. The 
recorded interviews were kept until the end of the study. After all the interviews were done, the researchers brought 
all the transcriptions together and co-worked on data analysis. 

Data Analysis 

The transcriptions of interviews were initially loaded into Atlasti for coding. The quotations and codes 
were then outputted into a Microsoft Word document and categorized into eight categories. Each researcher worked 
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individually on the output document to find emerging themes, then, they sat together to combine their individual 
findings for deeper analysis. The final findings are based on the agreements of all researchers. 

Use of Validity Strategies & Potential Threats to Validity 

To promote the validity of the research, researchers applied the following strategies: 

1 . Prolonged engagement and persistent observation: All researchers are current doctoral students and they all 
have experienced virtual teamwork. The researchers have been in the environment for a long period of 
time, and have built trust with research subjects. They are also very familiar with the culture of the 
environment. 

2. Triangulation: The research involves three investigators with different background and perspectives over 
the topic. 

3. Member checking: Researchers have shared interview transcripts and analytical thoughts to make sure they 
are representing the participants’ ideas correctly. 

In spite of the validity strategies that the researchers incorporated in the study, there might be potential bias 
that would possibly threaten the validity of the research since all researchers are studying in the same environment 
with the research subjects and may have built strong trust or distrust with the subjects. 

Results and Discussion 

Research findings will be reported under the following eight categories obtained from the available data. 
Guidance for virtual teamwork 

Generally speaking, students received very little guidance regarding how to work in a virtual team from the 
instructor. One respondent said: “We had from our instructors suggestions for how to meet or how often to meet, we 
got some tips, little, not anything.” Another one said: “There’s not been a formal unit or lesson on proper practices 
or best practices for teamwork.” 

In summary, guidelines mentioned by one or other respondents included how to use the chat room, how to 
make the team visible by creating a team homepage, how and how often to meet, and what communication tools to 
use. However these guidelines are far less than enough for students to have effective virtual teamwork. The even 
worse situation is that some respondents didn’t remember to get any guideline from the instructor or anyone else. 
Obviously, instructors neglected the importance of the instruction of teamwork guidelines to students and were 
reluctant to spend time on it. It is probably because instructors take it for granted that students have acquired 
sufficient skills for virtual teamwork, or because they think the guidelines are common senses, or because they don’t 
regard it as their responsibility. In addition, one respondent said: “We actually had the guidelines on how to work on 
teams, but we need to find it from Internet resources.” Compared to those receiving very few or no guidelines, 
student who are required to find the guidelines by themselves are in a better condition, for they are at least aware of 
the need to search for the guidance of teamwork. 

The findings of the study by Napier and Waters (2001) indicate that educators who teach online graduate 
courses and assign online teaching projects should consider integrating online teambuilding instruction into their 
coursework. So, instructors should implement teambuilding instruction which is beneficial for student teamwork. 

When asked about what guidelines to expect from instructors, one respondent expressed his desire to know 
something about conflict solving and hoped instructors could raise his awareness of the possible conflicts by 
showing some examples. 

I personally feel that the instructor should first of all tell us how to avoid the conflicts and divergent 
opinions in teamwork. If he can show us some relevant examples of the conflicts and their consequences, 
we may get ready for the conflicts that could happen later in our teamwork. 

Some respondents emphasized their expectation of instructors’ help with technology tools, which 
corresponds to the statement by Duarte and Snyder (1999) that teamwork training should also include technical 
support and technology training on the hardware and software used to support online teamwork. There were also 
some respondents who said they didn’t care about the teamwork guidance. Since not all the students need it, 
instructors could conduct optional teamwork training. 

Team formation 

Based on respondents’ reply, teams are usually formed in two ways: instructor’s assignment and student’s 
free choice. In most cases, students are allowed to choose their team members by themselves and some are prone to 
work with acquaintances or someone they have worked with before. If they don’t know anyone in the course, they 
would like to know something about the others before forming a team. It seems that some instructors would help 
them to get to know each other: 
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At the beginning of the course, the instructor suggested posting brief descriptions of ourselves especially 
related to strengths and weaknesses. So when you make a team, you have to get others who have strengths 
that you don’t have. So you make a balance in your team. 

In my mixed mode class, we had a face-to-face meeting at the first class. We saw each other and talked 
about our similarities and differences and tried to form a team. 

Requiring students to post a description of themselves with their strengths & weaknesses and setting a face- 
to-face meeting at the beginning are both good strategies to help students form a balanced team. In addition, it is 
very common among instructors to request students to introduce themselves on bulletin boards and respond to others 
at the beginning of the course. Some respondents supported the use of bulletin boards for getting acquainted with 
each other and considered mental contact more important than physical contact, while others still preferred a face-to- 
face meeting at the beginning: “It’s sometimes best to have them around and you can observe them and see them in 
a natural setting, then that will give you an idea what they work like.” 

Regardless of which strategies are adopted, instructors should provide students opportunities to know each 
other before asking them to choose their partners. As for the preference of team formation methods, one respondent 
said that she didn’t like random assignment: “In my opinion, it is not a good idea for the instructor to put students 
randomly into groups, because you don’t know what strengths each student has.” Some respondents articulated they 
did not have preference for a particular team formation method, because both random assignment and free choice 
had their own advantages and disadvantages. 

People get a habit of working with the same people. I would like to mix up the groups sometimes, so I think 
the only way this might happen is we were assigned. So it might be ok to be assigned sometimes. But I like 
being able to choose. 

I like random assignment, because there are students from different countries in the school, but sometimes 
American students would more like to work with American students and Chinese students will prefer to 
work with other Chinese students. 

Apparently, random assignment ensures group diversity and enables students of different ethnicities to 
work together. But at the same time, respondents expressed their concern of being assigned to a “bad” team. When 
doing team assignments, instructors should try to make a balanced team based on students’ backgrounds. One 
respondent shared her favorable experience of being assigned to a team: “In that class we were assigned to teams. 

We had students of different years in the program. And she tried to pair some of the more experienced team 
members with some of the less experience students.” 

Team size usually depends on the project. For the project of a course, most respondents said 3 or 4 was 
their ideal number of people in a team, because larger size would lead to more conflicts and smaller group would not 
have enough collaborative thoughts. 

Communication 

The collaborative tools used by respondents include email, discussion board, chat room, instant messenger, 
and telephone. Just like one respondent said, every tool is suitable for certain tasks: 

I think email is good for just sending information. And bulletin board is good for response because it allows 
you to see multiple people’s responses. Chat room is good for discussion. So I think each form of 
communication has its value. 

Although a variety of tools are used, asynchronous tools such as email and bulletin board play a dominant 
role in students’ distance communication. 

In spite of the fact that a lot of communication tools are available, students still feel the necessity of 
occasional face-to-face meeting when the task is difficult or has technology components: 

It depends on the difficulty of the project. One of the assignments was a statistic class that requires use of 
software, the SPSS software, so we’d usually meet to put together the actual paper aspect and that was 
more involved project. When there was a smaller project, we didn’t meet face to face. But I guess for a 
large project. 

Additionally, some respondents had face-to-face meeting sometimes because of the downsides of email: 
Sometimes, the information that you try to convey via email or other electronic tools will deviate from 
what you really want to say. The interpretation of certain words will vary from person to person. 

I think sometimes email cannot convey like emotions, or can’t convey like expectations. So I think that the 
very first face-to-face meeting, even if it’s the only one is very vital. We level the ground so we are all on 
the same level so we can work from there. 

Aside from individual preferences of meeting face to face, when the problems can not be solved solely by 
written communication, none of the respondents mentioned the use of synchronous audio/visual communication 
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tools. The needs of having face-to-face meetings could be a result of lacking the capability and skills of using such 
tools. 

Team Leader and Roles 

In our interviews, all of the participants agreed that there should be a team leader to guide the process along. 
The team leader was identified as someone who can assign the roles, initiate the discussion, mediate the conflicts, 
remind the deadlines and compile each member’s work to get the whole project done. Respondents expressed the 
necessity of the team leaders and a good leader “makes an effective team”: 

I think there should be a team leader who will be in charge of assigning tasks, initiating discussion. If there 
is no team leader, team members will have different opinions and may have conflicts. 

I think in classroom setting, it helps to guide the process along and probably it is because we so used to 
have an instructor. When you get into a team, you are still looking for some person to play the role. 
Respondents also noted that in virtual teams “the role of team leader is more dynamic, the responsibility 
rotate, each member will lead on one of the assignments.” In their responses to the way the team leaders were 
selected, four of the nine respondents claimed that the team leader emerged during the process; three students 
reported that their team leaders were volunteers; and two interviewees said they took their turn to be the team leader. 

Team Conflicts 

Based on our literature review, a lot of studies suggested that, with no or limited face-to-face interactions, 
students in virtual teams were more likely to get into ambiguity. In order to better understand the students’ social 
interaction, respondents were asked questions regarding if they ever experienced any team conflicts, how they 
usually come to a consensus, and how they deal with slackers. 

From the answers of the respondents, a few common team conflicts arise: different opinions, blaming each 
other, hard to get consensus and, lagging behind or no contribution to the teamwork. In response to the question on 
how they come to a consensus, the interviewees addressed the following solutions: “go to the most popular one”; 
“take advantages of everyone’s strength”; and “compare pros and cons.” One of the interviewees mentioned a very 
good point: 

People have different strengths; we need to get use of this kind of strength. If someone is very good at web 
design, let him do your team presentation page; if someone is very good at editing, let him go through the 
final copies; if someone is very good at doing research, let him guide the research process. Let people feel 
they are very unique in the team and they do some contributions. 

A slacker is identified as one of the team conflicts. Slacker refers to someone who does not contribute to 
the team project and counts on others to complete assignments. Some interviewees addressed their solutions to this 
problem: 

When you have some person lagging behind or did not work on the project, you need to contact him/her 
using a nurturing way, email or call them and say ‘what’s going on?’ ‘Are you OK?’ Try to get them to 
work with you. Let them feel as a part of the team. They need to be concerned with getting to the work 
done. 

Some respondents mentioned that they “would let the team leader talk to the person nicely, if still no 
response, (they) will make up that work for him/her.” 

Based on the interview responses, it was found that team conflicts did not arise from task difficulty but 
usually from a lack of motivation to participate, inappropriate communications, and different opinions. 

Feedback 

Virtual teams have unique features compared to traditional teams in terms of the team interaction. Team 
members collaborate from a distance through technology. Research suggested that instructors providing constructive 
feedback to students have a positive impact on students learning outcomes and problem solving skills (Bjorklund, 
Parente, Sathianathan, 2002). In virtual teams, students are told to work together on their assignments or projects. 
However, whether or not collaborative learning actually occurred depends on what happened in the team. In online 
collaborative learning environment, the role of instructor is more likely to be the facilitator. In order to determine the 
relationship between the feedback of the instructor and the students team work performance, we asked students 
about their experiences of their instructor’s feedback on their teamwork. The response shows that the feedback from 
instructors is helpful on their product, but there is lack of the feedback on the team process (how well the team work 
together). The students’ responses for this question are listed below: 
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I think feedback is really helpful to us to understand the project and assignment. Normally instructors 
provided feedback through email or rubrics. With rubrics, they will highlight the aspects that they think 
team had met and at the bottom they provided comments regarding the overall project. 

“Generally, the instructor seems to create one template to use on all of the teams. Adding points or 
removing points to fit each team, to personalize each team report.” 

“The virtual teams I worked with, instructor gives us some feedback which is helpful for our product, but 
he did not give us feedback on the teams.” 

“Most of time, I think instructors go through the whole group work but they don’t know much about each 
member’s contribution.” 

When researchers asked the question; “how would you like the feedback to be.” one student said she really 
liked constructive feedback which instructor addressed problems and specific points on how to improve the project. 
Another student mentioned he liked the way that the instructor met with each team member to talk about the project 
and got to know what’s going on. 

It appears that students have realized the importance of the feedback from the instructor and instructors 
should provide feedback on two levels- content (product quality) and team performance (how well the team worked 
together). 

Comparison with Face-to-Face Teamwork 

On one side, compared to traditional face-to-face teams, virtual teams can be more effective and efficient in 
some situations because of the flexibility and timeliness of online communication, but on the other side, 
communicating through instant messengers or emails can be extremely ineffective. One interviewee specifically 
pointed out that “with face-to-face we can brainstorm and within an hour we could have more than what we can get 
from chatting online for three hours.” Most of the interviewees did agree that the overall virtual teamwork 
effectiveness depends on the types of the project and the personalities, skills, and experiences of the team members. 
Although technologies such as synchronous audio/visual communication tools might remediate this situation, 
acquiring the technology and skills of using the technology is a challenge to team members with different technical 
background. 

General Perspectives on Virtual Teamwork 

Although people’s perspectives toward virtual teams differ from one to another, some of them have found 
working in virtual teams “rewarding” and “more productive”. It “has always been a good experience”, team 
members can “get more things done” and “get very quick response”. Even those who “generally do not like them” 
acknowledge that there are “practical benefits of having virtual teams” because of the flexibility that allows them to 
meet with their teammates anywhere and anytime. It is especially helpful around assignment’s deadline. Team 
members who have difficulties to meet frequently can “talk every night” over instant messenger “when the deadline 
approaches soon” so that they can finish it on time. 

There are also some issues that most of the interviewees experienced in virtual teams. One of the problems 
or frustrations that they often have is with team members who lack technical ability. It is also hard to work with 
somebody that you never met before and to establish personal connection through online communication due to the 
lack of facial expression and body languages. It is suggested that web cam and video conferencing can be used to 
complement instant messaging and emailing to convey non-verbal cues and promote personal connection. 

Conclusions 

This study pinpointed some problems hindering the smooth flow of virtual teamwork and brought forward 
some best practices for instructors as well as students summarized as below. 

The problems identified with students in virtual teamwork include: 1) a lack of motivation to participate on 
virtual teamwork, 2) a lack of specific guidelines on technology use and how to collaborate in virtual teams, 3) 
limited use of synchronous, audio, visual communication tools, and 4) difficulty in establishing personal connection 
via text-based tools such as email and instant messenger. 

Some of the best practices and strategies for students encompass: 1) keep the technology in align with 
project and team needs; 2) clearly identify the team roles and timeline for the project; 3) respect individual team 
members; 4) shift the role of team leader to each member who has different strengths; 5) communicate with 
members frequently and clearly; 6) approach as a team and kindly remind the person who is not contributing. 

Best practices and strategies for instructors generated from this study consist of : 1) Conduct optional 
teamwork training including technology instruction; 2) Encourage students to work with different people; 3) Take 
students’ background information into consideration when assigning students into teams; 4) Assign students into 
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teams of 3 or 4; 5) Provide opportunities for students to know each other before team formation; 6) Encourage 
students to use audio and visual communication tools; 7) Provide team member evaluation forms to encourage 
students involvement in teamwork; 8) Keep an eye on each team’s progress to facilitate and motivate students 
learning in virtual teams. 

The outcomes of this study have very practical significance. Those best practices for instructors and 
students will definitely improve the effectiveness of virtual teamwork. 

Future Research 

“Virtual teamwork” in this study refers to students’ teamwork in traditional face-to-face class, mixed mode 
and totally online class as long as it involves online collaboration. Future research may just focus on virtual 
teamwork in totally online course. 

Among the 9 participants of this study, the majority of them are Americans and 2 are Asians. The data 
collected indicate they have somewhat different opinions over a few aspects of virtual teamwork. Future research 
study could compare the differences between Americans and students from other countries in virtual teamwork. 

Additionally, this study found that students lack the motivation to participate in virtual teamwork. In future 
research, researchers may search for the reasons behind it. 
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Mathematics is central to many academic disciplines: engineering, physics, and statistics to name 
but a few broad categories. Mathematics has even been described as the language of science (Dantzig, 
2005). Despite the importance of communicating mathematics, no universally accepted method for 
authoring or displaying mathematical expressions for the World Wide Web (WWW) has emerged. This 
fact is somewhat ironic given that the WWW’s origin was at the European Organization for Nuclear 
Research, CERN, the world’s largest particle physics laboratory and one of the most respected scientific 
research centers (CERN, 2008). 

The lack of a universal method of displaying mathematics on the WWW has consequences for 
developers, learners, and professionals who need to communicate mathematical content. Developers must 
select an authoring and delivery technology that may disenfranchise potential users either because of their 
choice of computer platform or disability. Learners attempting to communicate mathematics electronically 
are left to use ad hoc methods of representing mathematics that may change from course to course. Eor 
example, keyboard characters can be used to represent exponents like x'^2 representing the variable x to the 
second power. Some expressions can easily be communicated in text such as x to the second power. But 
these methods become cumbersome as the mathematical expressions increase in complexity. The lack of a 
good method for the display mathematics on the WWW has been cited as a hindrance to more widespread 
use of online journals in mathematics (Jackson, 2000), and the issue of displaying mathematics on the 
World Wide Web has been listed as a key issue by The Math Eorum, an Internet center for mathematics and 
mathematics education operated by Drexel University’s School of Education (MathEorum, 2008). 

A literature search for methods of displaying mathematical expressions on the WWW produced a 
modest amount of information. Some efforts (Majewski, 1999; Siegrist, 2006; Zhi-Eeng, Liu, Yuan, & Lo, 
2004) have been made to catalog the various methods possible for displaying mathematical expressions on 
the WWW, but no studies found in the review of the literature have recognized widely accepted or standard 
methods. MathML ("W3C Math Home," 2007) has existed as a recommendation from the World Wide 
Web Consortium for encoding mathematical content since 1998. By technology standards that makes it 
relatively old, yet widespread support for displaying documents utilizing MathML has not yet been 
realized. One obstacle to the use of MathML is that it is considered difficult to author with it (Bringslid, 
2002; Majewski, 1999; Siegrist, 2006). Also, not all web browsers will display MathML content. 

The purpose of this research project was to determine the tools and methods currently used in the 
development of web-based instructional materials containing mathematical expressions. Specifically, this 
research was conducted to answer the following research questions: 

1 . Is there any one tool, process, or technology used pervasively by mathematics professionals in 
higher education to develop mathematical content for the World Wide Web? 

2. What barriers exist for developers and end-users that prevent the communication of 
mathematics on the World Wide Web? 

Method 

This research was conducted using a web-based survey with forced-response and open-ended 
questions. The survey used in this study was constructed after consulting the limited research on this topic. 
It was then distributed to five professionals with experience in developing mathematical content for the 
WWW. The survey was modified for both content and usability based on the feedback from this panel of 
expert reviewers. 

The final survey consisted of five sections: demographics, teaching/instruction, research, 
accessibility, and conclusions. The demographics section used mostly forced-response items to collect data 
on the participants’ background, work environment, computer use, etc. The teaching/instruction and 
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research sections asked participants to list the tools they use to author mathematical content for delivery on 
the WWW in each context. They also were asked to provide a description of the process they use to create 
their web-based instruction and research materials containing mathematical expressions. In the 
accessibility portion of the survey, participants were asked questions about the accessibility of their web- 
based materials and their knowledge of laws regarding accessibility of web-based content. The survey 
concluded by asking the participants to elaborate, if needed, on any of their forced-response answers and to 
provide comments they felt relevant to the topic or the survey. 

The survey was distributed during a two-month period in the summer of 2007. The survey was 
promoted on the website of the Journal of Online Mathematics and its Applications. In addition, 200 
mathematicians from across the globe, who were working in various levels of higher education institutions, 
were sent an email invitation to complete the survey. The initial invitation requested that the email be 
forwarded to anyone with the necessary experience to contribute meaningfully to the study. The end result 
was 73 completed survey entries. Additionally, six participants chose to send comments via email to the 
researcher rather than responding to the survey. The majority of participants were working in the United 
States, but responses were received from participants working in Canada, Australia, France, and the United 
Arab Emirates. 



Results 

The survey responses were analyzed using content analysis (Merriam, 1998, p. 159; Patton, 2002, 
p. 453). The researcher organized the data for analysis, read through all of the responses making initial 
codes, classified all of the initial codes into categories, interpreted the code categories, and identified core 
consistencies. The results of this analysis are organized by the research questions investigated. 

Tools and Technologies 

In response to the research question “Is there any one tool, process, or technology used 
pervasively by mathematics professionals in higher education to develop mathematical content for the 
World Wide Web?” the content analysis of the survey responses resulted in the categories: (a) Portable 
Document Format (PDF), (b) FaTeX/TeX, (c) Embedded HTMF, and (d) MathMF. 

There was a clear preference for creating PDF files. Over half of the participants use software to 
generate PDF files of their mathematical content. Not all participants indicating the use of PDF files listed 
the software used to generate the mathematical content, which was then printed in PDF form. However, 
the majority of those who provided detailed descriptions of their process listed FaTeX or TeX (Knuth, 
1984) as their preferred mathematical typesetting tool. One participant unknowingly summarized the 
feelings of the majority with the statement, “the mother of all math-display software is TeX”. After PDF 
usage, the second most common method of communicating mathematical expressions on the WWW was 
the use of embedded graphic images in HTMF documents. Again, the tool most often used to create the 
HTMF with embedded images was TeX or FaTeX followed by a utility program, latex2html 
(latex2html.org, 2008). MathMF was listed a choice by 10 participants, followed by 9 who used Microsoft 
Word and its equation editor, or MathType ("MathType," 2008) add-on. All but one of the MathMF users 
listed that they used several different tools and technologies including PDF files. 

Barriers to Displaying Mathematics on the WWW 

The second research question, “What barriers exist for developers and end-users that prevent the 
trouble-free communication of mathematics on the World Wide Web?” was investigated by asking 
participants about other tools and technologies for displaying mathematics on the WWW as well as issues 
of accessibility. The content analysis resulted in response categories: (a) MathMF curiosity, (b) Cost of 
change, and (c) accessibility uncertainty. 

Participants were asked if they knew of better tools than the ones they were using, and if so, why 
they were not using them. Responses indicated that other than the tools being used, users were unaware of 
what they would consider better methods. Several participants listed a curiosity about MathMF, but stated 
concerns of how usable MathMF content would be by their audience. Such factors as problematic browser 
compatibility and difficulty with authoring MathMF were listed. One participant expressed 
disappointment with MathMF responding, “XMF-based systems, such as MathMF, don't seem to have 
lived up to their promises.” Of course, the cost in terms of time to learn new tools and the monetary cost 
associated with acquiring new tools also were listed as inhibiting factors. 
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The accessibility to web-based learning materials to individuals with disabilities is an important 
issue. Participants were asked if the online content they develop is accessible to visually challenged users. 
Only 7 participants responded that their materials were accessible to the visually challenged. Most 
participants either did not know if their materials were accessible or responded that they are not. 
Additionally, participants were asked if laws exist regarding the accessibility of their online content to 
individuals with disabilities. Only sixteen participants responded yes to this question. Of those responding 
no to this question, thirty-eight were working in the United States. 

Discussion 

The importance of TeX and LaTeX to creating documents containing mathematics is part of the 
culture of mathematicians. Their dissertations and journal articles are typically required to be typeset using 
one of these tools. The findings of this study offer confirmatory evidence to Majewski’s (1999, p. 142) 
assertion that the TeX and LaTeX software packages are the standard in scientific publishing. A participant 
commented, “It is inconceivable that, in the current state of web tools, I would be able to display my 
research without using TeX.” Developers often fall back to a lowest common denominator such as PDF 
files or embedded graphic images to properly represent complex mathematical symbols. This is not an 
optimum solution as stated by one participant, “Of course, linking to a PDF file is a little lame. However, I 
do not want to take a risk that somebody cannot view it correctly.” Given the evident user base of TeX and 
LaTeX to create mathematics, any tool developed to create mathematical content for web delivery should 
use this as the input source, if there is a desire to attract a large number of adopters. 

Many tools exist which allow authoring with TeX or LaTeX, but there are significant issues 
viewing such documents with a web browser. TeX and LaTeX cannot be natively displayed in any 
common web browser and alternatives such as MathML require proprietary plug-ins that are not available 
for all computer systems or require only certain versions of specific web browsers. 

The static nature of the use of PDF files or embedded graphic images, and the clumsy nature of the 
necessary creation of such formats, typically inhibits online collaboration and synchronous communication 
involving mathematics. Siegrist (2006) cautioned against the use of PDF and proprietary tools commenting: 
“Worst of all, much of the inherent interactivity of the web is lost with these formats.” (The Basic 
Document, paragraph 4). Both Siegrist (2006) and Majewski (1999) have detailed lists of additional 
drawbacks of these methods. Additionally, significant accessibility hurdles are created by these formats for 
any user who is visually challenged. Miner (2005) addressed the legal accessibility requirements of 
mathematical expressions well when he wrote: 

“Currently, in the case of Mathematics, these requirements are typically met by providing a text 
equivalent for equations. For example, in an HTML page, this typically means equations are 
displayed using images, with a textual ALT description of an image. Unfortunately, text 
descriptions of mathematical expressions only meet the letter of the law and do not really address 
user needs. At a practical level, the preparation of text descriptions is labor intensive and error 
prone. At a deeper level, for audio rendering of mathematics, the ability to “navigate around a long 
expression is critical to comprehension. Moreover, static text cannot take advantage of locale or 
user preference information to choose the language or customize the vocabulary.” (p. 537). 

There are technologies and tools that solve some of the problems associated with displaying 
mathematical expressions on the WWW. A project called jsMath (Cervone, 2008) very cleverly uses 
JavaScript and style sheets to enable authors to include LaTeX mathematics in their web pages and it is 
rendered quite well by many common browsers. jsMath does require the user to install a few free font 
packages. The use of jsMath simplifies the workflow for developers and removes many of the viewing 
issues for end-users. No proprietary plug-in is needed and it works across many computer platforms and 
web browsers. The content created by jsMath also scales to large sizes with virtually no loss of readability, 
hence some visually challenged individuals will be able to enlarge the type with no special software 
needed. Still, more attention needs to be given to the accessibility issues of mathematical content. 
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Learning at least a small set of TeX or LaTeX syntax is not a difficult task. If jsMath, or a similar 
technology, could be established as the standard for displaying mathematics on the WWW, there could be 
an explosion of communication and online collaboration possibilities involving mathematics. Accessibility 
beyond being able to increase the size of the type of on-screen documents is still an issue that needs 
addressed. T.V. Raman (1994) developed AsTeR, an audio system for technical readings that produces 
audio from electronic documents including documents containing LaTeX elements. A combination of 
jsMath and AsTeR could be a solution to some of the problems discussed in this paper. 

This research project was designed to document the tools and methods currently used by 
mathematics professionals for the display of mathematical expressions on the WWW. The solutions for the 
display of mathematical expressions presented in this discussion have not addressed other important issues 
such as the readability of the content embedded in web pages. For example, MathML efforts are divided in 
presentation MathML and content MathML. Content MathML is intended to allow for the meaning of 
mathematical expressions to be interpreted by software agents and to allow for meaningful searching of the 
mathematical content in web pages. As the Semantic Web (W3C, 2008) evolves, MathML may gain a 
larger user base than it has been able to secure since its first release in 1998. 
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Abstract 

This writing presents empirical research using the Torrance Test of Creative Thinking to explore creativity and its 
relationship to circadian rhythms. Studies involved first year college students in design. Separate studies examined 
impacts of training on creativity and the variation of creativity at different times during the day. In one study, pre- 
and post- treatment evaluation showed improvement in measured creativity due to training. The second used 
repeated measures at different times of day to show significant differences, indicating a circadian influence on 
creative capability. Creativity, as a higher order cognitive function, appears to be affected by biological factors. 

Introduction 

Creativity, is quantifiable and can also be qualitatively evaluated for divergence, the internal and external 
differences of their ideas. Creativity is definable, measurable, and open for evaluation; measurement of creative 
actions provides a means to understand ways of training and paths to improvement in creativity. Creativity can be 
taught, developed, encouraged, and improved. Creativity requires cognitive effort which may be affected by context 
and abilities, including nutrition, social environment, and by the cycles of night and day. 

Creativity, the development and adoption of new ideas is an important element in any domain. That skill is 
important at all levels of activity, from initial idea to finished product, and from nano scale inventions to city 
planning. Governments and businesses alike find value in increasing creativity. 

Training people to be more creative is effective, and long term changes in ability are achievable. While people can 
become more creative, it is important to recognize that social, biological, and cultural structures can affect creativity 
as it is practiced (See, for example Puccio, 2006 and Amabile, 2003). 

While substantial research has occurred on the training, development, and social contexts of creativity, less 
investigation has occurred as to the environmental aspects of creativity. Being creative requires substantial cognitive 
involvement, and elements that have an impact on the ability to think must be understood. Specifically, the time of 
day may have a significant impact on cognitive ability. 

Here we will begin with an examination of standard aspects of creativity; definition, applicability and it's 
relationship to innovation and intelligence. The generation of new ideas is the most essential element of creativity 
which is measured by the Torrance Tests of Creative Thinking. Given a large data base of previous results, this 
standardized test allows comparisons with a wider population for creativity norms. Two studies using the TTCT that 
examine creativity and circadian rhythms will be presented followed by a brief examination of the environmental 
and personal impacts on creativity. 

Creativity and intelligence 

While there may be many popular explanations and examples of creativity, the generally accepted functioning 
definition of creativity is the cognitive ability to generate novel and applicable ideas. "Creativity is the generation of 
new ideas — either new ways of looking at existing problems, or of seeing new opportunities..." (Cox, 2005, 8). "The 
creative process... refers to the sequence of thoughts and actions that leads to novel, adaptive productions" (Lubart, 
2001). 

Innovation is often used as a synonym for creativity, but it remains distinct from creativity: " ‘Innovation ’ is the 
successful exploitation of new ideas. It is the process that carries them through to new products, new services, new 
ways of running the business or even new ways of doing business" (Cox, 2005, 8). 

Our focus here is on creativity, which could be described as the spark to the fire of innovation. Innovation is 
concerned with the adoption and use of new and different ideas. As fire needs air and fuel, so too does the 
innovation need social support, resources, and change (Florida 2002, Rogers, 1991); but still, metaphorically, the 
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igniting spark is provided by creativity. One can be creative without being innovative; one can have a spark without 
ignition. Not all ideas catch fire. 

Creativity can be developed through a number of methods including motivational, cognitive, and social approaches 
(Bull et. ah, 1995). This writing assumes some creative ability in all individuals and that training is effective in 
increasing measured creativity. 

The Research Venue 

This research was conducted in two regular offerings of a large introductory lecture course on design. Each study 
occurred over a separate semester of the course, and they were used to gather data on creativity from a large 
population of design students. The first study compared a control group with an embedded group receiving creativity 
training through a separate course. (Specific instruction in creativity was not included in the larger course.) The 
second study examined creativity at different times of day. 

The smaller separate creativity course was a blend of theoretical instruction, application, and rapid idea generation, 
and was taught by the author. The nature of the course was consistent with recommendations inherent in Fasko 
(2001), and the findings in Scott, et. al. (2004). 

Scott, et. al. (2004) said that "divergent thinking" was common in most training efforts for creativity. Divergent 
thinking is the development of multiple responses to questions, and the ability to provide numerous applicable 
answers. 

Scott, et. al. (2004) completed a meta-analysis of 70 studies of creativity training. A number of elements 
differentiated results. Time on task and extensive work were usually needed to develop skills in creativity. Courses 
that focused on specific structured techniques were usually more effective than courses that merely dealt with open 
ended creative exercises. The largest gains in measured creativity occurred by the use of these structured techniques, 
such as convergent thinking (focusing ideas on a given result), critical thinking (thinking about one's thought 
processes), and the identification of problem constraints. 

Measuring creativity 

Different psychometric measures of idea generation were developed and popularized by Guilford and Torrance, 
early researchers in the field of creativity. This research used the Torrance Tests of Creative Thinking (1974a), 
which has been described as "... by far the most commonly used test of divergent thinking and [which] continues to 
enjoy widespread international use" (Plucker & Renzulli, 1999, 39). 

As with any testing method, there are limitations in validity and pragmatic concerns. The TTCT provides a good 
understanding of some aspects of creative abilities, focusing on the development of new ideas. The test can easily 
administered and can be scored by either the researcher or publisher, and the results can be compared with a large 
historical population. 

The Torrance Tests center on the idea that creativity can be measured through generative output, that is, ideas set 
forth in response to stimuli. We understand that creativity can be enhanced through training and other educational 
factors. In addition, creativity, as a higher order cognitive function, is affected by context and biological factors. 

There are two forms of the Torrance Tests (TTCT), figural and written. The written version was used in this 
research; it contains six sections in the test that ask for multiple written responses to illustrations and verbal prompts. 
Responses are sought over five or ten minutes. In addition, there are two full versions of the test which are designed 
to complement each other and be used in any order. The two versions were used as repeated measures. 

Within the Torrance written tests, there are three scoring areas that provide detail on creativity; fluency, flexibility, 
and originality. Fluency is based on the understanding that creative people generate more ideas than non-creative 
people. The Torrance Tests pose a hypothetical situation and ask for responses. Participants are evaluated by the 
number of answers given. For example, if question asked what one could eat, responses could include, for example, 
pizza, a salad, or fish. Providing more answers would be indicative of a higher level of creativity. 
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Flexibility is a measure of how the answers of an individual vary among themselves. Creativity is a skill that 
focuses on the development of new and different answers; generating slight variations of one theme would not be as 
creative as generating different types of answers. Building on our previous example, minor changes in a food type 
would not be considered a different answer; different flavors of ice cream or different types of salad would not be 
considered different answers. 

The Torrance measure of Originality compares answers to a common set of responses to the same question. In our 
example "salad" is an answer that could be received in many places around the world, but eating "crow" or one's 
own words would be less expected responses in any culture. New, novel, and unexpected answers outside of societal 
norms are, by definition, more creative. 

Testing 

The venue for the testing was a large lecture class on design thinking. This class is required of all new design 
students at the university and also includes students from other areas of study as well. It consists of lectures, 
readings, and projects dealing with design. While all students in the class took the TTCT, in this study, only design 
students were scored and evaluated. 

The test was administered twice over the semester, separated by about 14 weeks. 53 first year students were scored 
for this study, and were selected because they were majoring in design. 46 of the participants were female, and 7 
were male, a gender breakdown that is generally consistent with design students registered at the university. 

Scholastic Testing Services, publisher of the Torrance Tests was used for test scoring to insure reliability and 
accuracy. 

Results 

There was no measured gain in creativity by students in the lecture course between the two testings. The fluency 
score increased by 2.5%, an insignificant change. The score for flexibility increased by 2.9%, and this change was 
also not significant. In contrast, the originality score increased by 16% which was statistically significant. 



Table 1: Scores and comparisons 





RS - A mean 


RS - B mean 


Difference 


t-test A:B 


Fluency 

Control [n = 5 1 ] 

[SD] 


88.40 

[27.08] 


90.56 

[33.04] 


102.45% 


0.49 


Fluency 

Treatment [n = 9] 

[SD] 


94.67 

[26.81] 


109 


125.44 

[33.73] 


122 


Flexibility 

Control [n = 5 1 ] 

[SD] 


49.40 

[11.16] 


50.84 

[12.77] 


102.92% 


0.36 


Flexibility 


48.33 




63.89 




Treatment [n = 9] 




111 




133 


[SD] 


[10.84] 




[10.29] 




Originality 


59.19 


66.16 


116.35% 


0.017* 


Control [n = 5 1 ] 


[22.73] 


[26.60] 






[SD] 










Originality 


66.00 




98.56 




Treatment [n = 9] 




116 




124 


[SD] 


[23.41] 




[31.87] 





RS = raw score. SD = standard deviation. * = significant at .05 
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Scores of students within this group that received training significantly increased their scores for fluency, flexibility, 
and originality by over 30%. There was not significant difference between the treatment and control groups prior to 
treatment; after treatment the training group had measured significant comparable gains (Hokanson, 2007). 

A side order: Breakfast 

A few additional contextual elements were informally examined during the first study. It had been hypothesized that 
gender, dominant-handedness, and nutrition could have an impact on creative performance as evidenced by the 
Torrance Tests. As other research had indicated that cognitive abilities are influenced by similar factors, a series of 
adjunct questions were added to the TTCT. Specifically, participants were asked if they had had breakfast prior to 
the 930am testing. 

Breakfast, for the purposes of this study, was defined as "something more than coffee". 27 of 51 participants said 
they had eaten breakfast prior to research, and no data was gathered on food intake. On all three measures, those 
students eating breakfast scored significantly higher than those who did not. This informal investigation clearly hints 
at a direction of future research, and indirectly led to the research on creativity at different times of the day. 

A study of circadian rhythms and creativity 

The Torrance Test of Creativity was also used to investigate the influence of time of day on creativity. 

Circadian rhythms, the daily rhythmic changes of a 24 hour day, affect a variety of biological functions. While 
most people are familiar with larger seasonal effects due to changes in length of day, the daily changes of circadian 
rhythm are more subtle or noticeable. Jet lag is the most well known disruption of circadian rhythm. 

Humans, like most life forms, have a regular, daily schedule of biological rhythms. The most evident cycle is the 
sleep-wake cycle, but over 24 hours, heart rate, metabolism, urine production, cognition, body temperature, and 
receptiveness to certain medicines all vary. 

Research into circadian rhythms began with body temperature measurements on subjects isolated from daylight in 
underground caves by research/subject Norman Kleitman (1939). He found that body temperature, which may be 
connected with cognitive activity reached a low point early in the morning (bam) and peaked in early evening. 

In subsequent research, cognitive capability has been found to be at its lowest on waking in the morning through the 
use of dexterity, mathematics, and drawing to evaluate mental skills over a normal daily cycle (Kreitzman, 1963). 
Performance generally peaks at noon or shortly thereafter, declining gradually, with some variation until late 
evening. Colquhoun (1972) also found a daily rhythm of mental ability. He held that changes in body temperature 
paralleled mental ability, and was not causative, per Kleitman (1939). (see also Folkard, 1990). 

...rhythms include our body temperature, which is higher during the day than at night, as is or 
heart beat and blood pressure; we have a clear sleep/wake cycle; our cognitive ability change 
rhythmically over a 24-hour period. Tooth pain is lowest after lunch; proof reading and sprint 
swimming are best performed in the evening; labour pains more often begin at night and most 
natural births occur in the early hours; sudden cardiac death is more likely in the morning (Foster 
& Kreitzman, 1005, 11). 

Folkard found that mental ability is dependent on a range of variables including circadian time; it also is affected by 
waking hours (time awake) and type of task. Some tasks, such as those involving short term memory are 
significantly better performed in the morning, while some tasks, such as editing are better undertaken in the early 
evening (Folkard, 1990). Creativity, as a relatively complex mental task, would be expected to follow the general 
curve of abilities of body temperature, and should be more evident later in the day. 

Within this study, the age of the study participants was anticipated to also affect the results of the study. Young 
adults have a daily circadian rhythm that is less defined than older adults and the general cycle occurs later in the 
day by 2-4 hours. This could mean that early morning cognitive testing of younger subjects could result in lower 
scores than with more mature adults. In response to this phenomena, many school districts in the United States have 
reacted pragmatically and changed start time for high school students (aged 14-18) to one to two hours later in the 
day. 
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Method and venue 

142 undergraduates were tested within a large lecture class using the verbal Torrance Test of Creativity at the 
regularly scheduled class time of 9:30am. Eight weeks later, volunteers, solicited through extra credit, were retested 
using the TTCT. 57 volunteers completed the test at 4 pm on one of two days; 56 participants had taken the test in 
the classroom morning session, and were scored for this study. The two different forms of the verbal TTCT were 
used for this study. 

As the biological daily clocks vary between young adults and more mature, 30-year-old adults, the age of the 
participants was an important concern. The average age of the participants was 19.6, and the age of the participants 
ranged from 19 to 29. It was hypothesized that this comparative youth would help reveal cognitive differences in 
retesting, and that the later testing would demonstrate gains due to circadian rhythms. As noted in the previous 
study, simple retesting, without treatment does not demonstrate significant gains, (e.g. 9:30 a.m. to 9:30 a.m.). 

In all areas, subsequent afternoon testing revealed significant differences in performance. See table 2. 



Table 2: Time of day scores an d comparisons 



Group 

[n=56] 


RS B 9:30am 

mean 


RS A 4:00pm 
mean 


Percent 

A/B 


t-test 

B:A 


SS-B 

mean 


SS-A 

mean 


Diff 
in SD 


Fluency 

[SD] 


73.94 

[21.34] 


114.86 

[28.64] 


155.34% 


0.000* 


102.57 


122.20 


19.63 


Flexibility 

[SD] 


39.90 

[8.17] 


60.96 

[10.74] 


152.79% 


0.000* 


99.61 


131.14 


31.75* 


Originality 

[SD] 


44.98 

[19.21] 


79.96 

[23.91] 


177.77% 


0.000* 


102.10 


127.02 


24.92* 



RS = raw score. SS = Standardized score SD = standard deviation. * = significant at .05 

While there were differences between areas of testing, the strength of change in each area was substantial. The mean 
for Fluency, the ability to generate a large number of answers to a prompt, increased by 55%, which was significant 
at .05. Standardized scores, a distillation of raw scores in comparison with the broader society, increased by 19.64. 
As the mean for TTCT standardized scores is 100 and the standard deviation is 20, this may not be significant. 

The mean scores for Flexibility and Originality both increased significantly. Flexibility increased by 53%; 

Flexibility is a measure of the divergence of individual participants' responses. Originality, the ability to generate 
uncommon responses increased by 78%. 

Discussion 

Time of day would appear to be an important consideration in the application of creativity, or with any other 
substantial cognitive effort. At the least, creativity is a challenging mental activity, and one which cannot be viewed 
as equally available every hour of the day. The mental effort of creativity needs to be treated as a sensitive aspect of 
the human mind, and specifically addressed with the resources of time and nutrition. 

This may mean that individuals should recognize their own personal circadian cycles and seek to work with their 
own inherent biological variances. 

These findings support the concept that challenges that involve creative and cognitive efforts must address the time 
of day when tests were administered. This could have an impact of forms of standardized testing such as college 
entrance or graduate exams. For example, this could mean that examinations directed at 18 year olds, such as the 
Scholastic Aptitude Test, should not be given at 8am. This and other research may also affect class and curricular 
scheduling of courses in creativity, design, and throughout the university. 

Those taking examinations at a standardized time may be more successful that if they were attempted at a time better 
suited to their own circadian rhythms. As is well known, each individual also has variances in their circadian cycle. 
During this research, each participant was asked about their own description as a "night" or "day" person. These are 
coarse measures of personal performance, and are only anecdotally accurate. In addition, given the complex 
variations found by Folkard (1979), there may be multiple peaks and troughs of cognitive ability during a 24 hour 
cycle. 
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The ability to perform specific cognitive tasks also varies throughout the day; e.g. as short term memory peaks 
during the early morning it may mean that didactic class sessions work well in the morning, which is consistent with 
many college lecture classes. Analytical tasks are better performed later in the day or early evening, which may 
indicate that those times are more applicable for active learning. At the least, student performance could be 
improved by lecturing in the morning and testing in the afternoon. 

This may also be a strong argument for online asynchronous learning. Students and faculty are both able to adjust 
their schedules to be more effective, in contrast with current practices, with wide-awake faculty and bleary-eyed 
students joined an Sam class. 

Conclusion 

This research measured creativity in two large groups of undergraduate design students. Students receiving training 
showed significant improvement of a large control group, demonstrating strong gains in all measured areas. In 
repeated measures, participants tested at different times of the day demonstrated significantly greater creative ability 
in afternoon examinations. 

It can be concluded that circadian rhythms can have significant effects on measured creativity, and it could be 
hypothesized that other cognitive functions would be similarly affected. Other biological impacts could also affect 
creative performance including recent nutrition. 
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Abstract 

This writing addresses the perception and activities of the instructional designer. At its core, this is an 
examination of beliefs, of values in the field, and of the questions we ask concerning aesthetics, innovation, and the 
very nature of design. Specifically, we present a theoretical and philosophical investigation of the perceptions we 
maintain of ourselves as designers. We begin by addressing the current state of instructional design, which, by many 
accounts, is limited by its approach to design issues. We believe most in the field function as and seek to be 
instructional engineers (Visscher-Voerman & Gustafson, 2004). The driving force in development of instructional 
materials is often the efficient production of instructional materials, an orientation which inherently limits 
innovation. This current practice in instructional design concentrates on the completion of technological and 
pedagogical requirements rather than quality of learning experience. We will introduce a new conceptualization of 
the processes and methodologies of instructional design, not to replace or supplant the traditional phases of ADDIE, 
but rather to illustrate a contemporary perspective with a refined collection of exemplars for design practice; the 
instructional artist, instructional architect, instructional engineer, and instructional craftsperson. As a sequence of 
inspirations, or even as a representative design team, this "role-based" design perspective is intended to encourage 
innovation in the instructional design field and promote higher quality design of learner experiences. 

Introduction 

Why has the field of instructional design failed to move forward? That is, forward with innovative ideas, 
forward with innovative methods of teaching and learning using technology, and forward with innovative theories of 
the use of educational technology? What we say as practitioners is important; the field of instructional design has 
frequently promised rich, authentic, effective learning through the use of technology. As instructional designers, and 
in terms of learning, however, we aren’t rich, we’re simple. We habitually find ourselves recreating the teaching and 
instructional methods of the past century using whatever recent technological innovations arise. 

Clark (1995) held that learning from instruction media would only change when the instructional method 
changed, regardless of media. In other words, using the same ways of instruction would always lead to the same 
results in learning. Media debate aside, our position is that use of the same methodology and processes for design 
typically leads to the same instructional products. Only limited success and innovation in design and research can be 
expected or achieved. Similarly, our designs will only change when our design methodology, perspectives, and 
values change, regardless of the medium of our work. In this paper, we introduce a new conceptualization of the 
processes and methodologies of instructional design, not to replace or supplant the traditional phases of ADDIE, but 
rather to illustrate a revised perspective with a collection of exemplars for design practice; the instructional artist, 
instructional architect, instructional engineer, and instructional craftsperson. We present these roles and their 
inherent values as a refreshing and encouraging perspective, potentially a metaphor, for instructional designers, both 
novice and expert, to reflect upon when critiquing our work and searching for avenues to advance innovation in the 
field. However, we must first examine the traditional instructional design methodology and discuss areas for 
progression, advancement, and expansion, specifically in our internal design perspectives. Challenging how we view 
ourselves, and how we can improve our own well-practiced skills can often be difficult. It has been our goal in this 
paper to stir discussion, controversy, and reflection among designers as some of the first steps toward advancement. 

The traditional design process 

Central to this temporary intermission of innovation in the instructional design field is our saluted design 
methodology. Before we continue, it is important to note that we are intentionally casting the field of instructional 
design as a “straw man’’ in our polemic argument for re-conceptualization of the traditional, often stagnant, 
processes and methodologies that have come to define our work. Most instructional designers, when questioned 
about their use of a design method, have a one word answer: “ADDIE.” Examining ADDIE does not provide an 
argument for its continued use. It is not an invented method for planning and designing in instructional design, but 
rather a codification of established and vernacular practice. It appears to be described in a number of sources as a 
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representation for what actually happens in many design fields, only later to be formalized through the literature of 
instructional design (c.f. Molenda, 2003). 

In common use, the ADDIE model differs very little from codified design models used in other fields, 
notably architecture. Architects commonly use the terms schematic design, design development, construction 
documents, and contract administration to contractually segment the process of design and constructing a building 
(see, for example, the AIA General Conditions of the Contract). However, while the process can be divided into 
these phases (along with subsequent client billings), within good architectural firms almost no designer limits or 
constrains their design activity to these steps. In reality, design still occurs during construction, often adding to the 
value of the work. 

One can view the ADDIE model as a recipe for instructional design, a perfunctory and mechanical 
description, or overview, of the design process. Novice cooks follow recipes, without modification or extrapolation, 
getting the expected result (c.f. Bird, 2007). Completion of the process, or of being done, is desired; done well is 
rare. The great cook, however, does not slavishly follow the recipe, but rather extends what is proscribed. Chefs, 
those we need to advance innovation and momentum in the culinary field, make use of their imagination, based on 
established processes and values, but also are not hindered by it. 

The process of design cannot be codified in a simple recipe in any sub-field, from graphic to urban design. 
Instructional design, although ostensibly following a single lockstep process, also exhibits this same diversity of 
methodology. In a study of instructional designers, Visscher-Voerman and Gustafson (2004) examined the design 
processes of instructional designers. Most followed a traditional, rational, ADDIE based model, but their research 
showed that "design processes are much more heterogeneous and diverse" than the ADDIE model suggests. 

Our argument holds that ADDIE, as a process through which instructional designers design, develop, and 
implement instructional media and learning environments, does little, if anything, to encourage further innovation 
when designing. We are not arguing for the exile of ADDIE, but rather a fresh perspective and set of values in 
support of innovation in the design of learning experiences. Orientation of the field toward pioneering design work 
that engages learners more meaningfully and effectively requires a focus on creating rich and innovative learning 
experiences, as opposed to simply developing instructional products through staid processes. We contend that to 
truly design is to extend understanding, to create something new and innovative; we believe design is a goal based 
on a set of values and philosophies, not a process. Advancing design innovation through use of a new set of design 
lenses and perspectives is the foundation of our argument. 

The use of proscribed methods in teaching instructional design is understandable. Providing learners with a 
set of procedures to follow that generates predictable results is, to some extent, valuable. Novices commonly seek a 
specific set of tasks that will guarantee success, using the model with fervor, evaluated on their loyal steps in the 
process, as (theoretically) consistent with design skill. In reality, this practice perpetuates a process of design that is 
lockstep in execution, removes creativity from much of the process, and isolates the engaging aspects of the work in 
the early stages. We feel it is a disservice to the novice instructional designer to present this method as the sole 
process of instructional design. Later aspects of the process are demotivating; implementation is viewed as simply 
getting the job done and/or drudgery; evaluation is an afterthought if remembered. However, as designers mature 
and address more complex problems, they do generally decrease use of specific sets of ordered actions and advance 
through evolution of internalized design values, perspectives, frameworks, and philosophies. Dreyfus and Dreyfus 
(1986) noted that novices in any field tend to seek rules and follow rule-based behavior; as an individual develops 
(design) skills, the value of a model decreases, knowledge becomes less tacit, rules less explicit, and capability less 
defined by declared knowledge. We contend that designers should be 'role-based' as opposed to 'rule-based'. 

Design is, of course, never done; good designers have in their psyche an impatience, a dissatisfaction with 
the status quo. They seek the challenge, the unexpected result. The goal of design education is not to produce 
consistent designs, particularly those that replicate previous work. The goal must be to produce better designers and 
hence, designs as yet unconceived. Inherent in this goal for the profession is one of constant improvement and 
innovation of the design process. Any design process has a series of different activities, with attention spent in 
various different aspects of the work. A broad understanding of the project, as well as background in the field and 
ability in each aspect of the work is needed. In addition, time must be dedicated to the experimental aspects of the 
work, both on specific projects and to generally advance. A rigorous understanding and evaluation of the field's 
body of knowledge must be applied. Ultimately, the project must be implemented with skill and continuous 
improvement of the design, even after formal completion. Seasoned instructional designers understand and execute 
this dynamic progression in their work. However, the process and methodologies we use to describe and define 
ourselves as a field, both to newcomers and alternative domains, do little to encourage the intricacies of the design 
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experience when truly pushing design forward. We feel our methodologies have lost, if not forgotten, the crucial 
values and perspectives that promote design innovation and advancement. 

A contemporary perspective for design innovation 

A number of theories and ideas guide the practice of instructional design; in addition, we are irrevocably 
guided by our perceptions of our own practice. How we describe the work of instructional design is how we do the 
work of instructional design. “As individuals express their life, so they are.” (Marx & Engels, 1845). For example, if 
we view the work of instructional design as the application of theories of cognition and learning, the inherent value 
is one of the learning sciences or perception. Similarly, if we view the process as simply one of the production of 
instructional materials, we may create work that is complete but limited in innovation. We must seek to include the 
full range of roles of the designer in every project, extending our self-image beyond that of the instructional engineer 
or manufacturer. 

One way to organize and present the values of instructional design is to use and emulate a series of roles as 
modes or exemplars for successful design practice. Being a designer, and acting as a designer, therefore, becomes 
more important than understanding what tasks a designer does, just as being creative is much more important (and 
difficult) than knowing what creativity is. Presented here is a contemporary perspective of instructional design, one 
which entails a number of "roles" rather than tasks. We acknowledge that good and experienced instructional 
designers encompass some or all of these roles in their work; however, when describing their profession or 
mentoring future designers, the inherent values, perspectives, and philosophies of each are often lost in translation. 
We believe there are roles that procedurally lead one through instructional design, and, more importantly, act as 
exemplars, which if followed, may help to improve quality and innovation within the field. 

Role-Based Design 

The roles of design we present here are archetypes, i.e., romanticized versions of real professions, 
exemplars of behavior and practice, personifications of values sets and philosophies, which are applicable 
perspectives for professional behavior in instructional design. As exemplars, we seek from them the best of their 
practices; for example, from the artist, creativity, and from the craftsperson, patience and advancement through 
practice. Each of the roles we have selected is well known through our society: artist, architect, engineer, and 
craftsman. The roles we highlight have been present in design for a long time, and most earlier design practices 
necessarily included all these attributes. In the Renaissance, these roles were blurred, integrated into the single 
individual or practice; for example, Leonardo da Vinci was artist and engineer, architect and craftsperson. Much of 
the instructional design field seeks the scientific, rational approach to design, where answers exist, and the best 
method can be found, adopted by all, then developed through completion. 

The two roles we believe are most present in current instructional design practice are the instructional 
engineer and the instructional manufacturer. Understanding the interplay between these roles and their innate 
perspectives describes traditional practice in the field. 

The Instructional Engineer (scientific realization) 

Engineering is the creative application of scientific principles used to plan, build, direct, guide, manage, or 
work on systems to maintain and improve our daily lives. While scientists explore nature in order to discover 
general principles, engineers apply established principles drawn from mathematics and science in order to develop 
economical solutions to technical problems. We use the term "instructional engineer" as an aspect of instructional 
design that is most addressed in instructional design programs. The instructional engineer focuses on applying 
research on learning. It is close to our vision of scientist, someone seeking new knowledge through research, but 
here, as the role is an applied one, the term "engineer" is most relevant. Indeed, some argue that the field of 
"instructional design" itself is mis-named: “Some object to the word ‘design,’ suggesting as it does a rather arty 
orientation, and insist that what we really need is ‘instructional engineering" (Shepard, 2002). 

In our perspective, the role of the instructional engineer is one of instructional problem solving. Most 
engineers, either in the instructional field or in the major sub-fields of engineering such as civil, structural, or 
mechanical engineering, are highly trained professionals. In education, the instructional engineer ensures a product 
is usable by the target audience and ensures the product achieve its educational goals. Contemporary, research-based 
ideas are used to develop instructional materials; educational theory is an important component of the work of the 
instructional engineer. The principle goal of the engineer is the functional efficiency of the work, planning and 
organizing the project. These are valuable aspects of the design process and can advance the value of the work. 

In current practice, most design completed by the instructional engineer is passed on to technicians with 
little opportunity for change. In any design project, at some point, the conceptualization, the planning, and the 
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broader view have been completed, and the work must be implemented. Here again there are significant questions, 
of a choice between completion and craft. Most instructional design work these days is manufactured, where ideas 
developed elsewhere are implemented by workers divorced from concept, aesthetics, or theory. 

The Instructional Manufacturer (efficient production) 

Instructional design materials are often produced by a manufacturer and not by an engineer. The 
manufacturer frequently is a technically skilled individual applying a pre-defined design template to solve an 
educational problem, delivering results as efficiently as possible. The solution to an educational problem is given or 
dictated to the manufacturer, whose responsibility is one of formatted production. Production consistency and 
stability are of primary value, resulting in products that are predicable and functional. As one expects a recipe from a 
cookbook to be predictably good but also what was intended, one should expect the results from a manufacturer to 
produce consistent, but not innovative work. For example, when asked to develop educational materials for use 
through distance education, the instructional manufacturer might employ traditional instructional design methods to 
develop instructional materials emphasizing content presentation and application. Such materials are commonly 
delivered to learners via the most efficient technologies (e.g., online quizzes, BlackboardAVebCT templates, 
PowerPoint presentations, etc.). Most of these technologies are stable and, at the core, are based on educational 
theories such as constructivism, collaboration, or cognitive science, but such theories are remote from the 
manufacturer. However, the role of the manufacturer is to implement the technology that best fits the dictated need, 
regardless of alignment with pedagogical theory. Models for the manufacturer’s design process primarily focus on 
the functional (i.e. "form follows function"). As with the architecture in the 1960's, an aesthetic could develop based 
on making the technology work, one based on utility. 

Speed and consistency are the values of the manufacturer; however, there are inherent problems with this 
simplified process. First, the experience and technical skill that may be present with the manufacturer seldom inform 
the design direction; design ceases with the conclusion of the engineering phase, and all prospects for qualitative 
improvement stop in a traditional manufacturing process. Second, criteria for success of the manufacturer are based 
upon quantity, not quality. While efficiency may improve, quality remains constant at most. 
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Figure 1: Current practice in instructional design. The manufacturer produces the design of the engineer; each 
project is separate, and does not advance the field. The quality of design does not improve. 



We seek to replace the role and inherent perspective of the instructional manufacturer with that of a 
craftsperson. The values of the craftsperson are critical to the quality of the end artifact; as part of the full design 
process. For the health of the design process and the participant designers, we argue that this portion of the work be 
positive, additive, generative, and ultimately forward thinking to ensure a gradual incline of quality in future 
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designs. At this point it is important to note that we do not view these roles as separate entities, but rather a unique 
collection of perspectives and values that an individual instructional designer, or team of designers, can bring to 
their design work. 

The Instructional Craftsperson (experienced evolution) 

The instructional craftsman encompasses the work of development and implementation, but also seeks to 
improve the project design. Traditionally, craft work implies a high level of skill in execution, and while not 
focusing on the research or theoretical foundations, there exists a solid theoretical understanding of the field. 

As a verb, 'to craft' seemingly means to participate skillfully in some small-scale process. This implies 
several things. First, it affirms that the results of involved work will still surpass the results of detached 
work. To craft is to care. Second, it suggests that partnerships with technology are better than autonomous 
technology. For example, personal mastery of open-ended software can take computers places that 
deterministic software code cannot. Third, to craft implies working at a personal scale— acting locally in 
reaction to anonymous, globalized, industrial production— hence its appeal in describing phenomena such as 
microbreweries. Finally, the usage of 'craft' as a verb evades the persistent stigma that has attached itself to 
the noun. (McCullough, 1998, p. 21) 

In modern society today, we have a view, a vision of "craftsperson,” one of a highly skilled trades worker 
creating exceptional work, a benevolent artisan. Historically, a craftsperson was a highly skilled guild member, 
required to mentor an apprentice to continue and advance the guild mission. Inherent in the role of master 
craftsperson was the requirement of building the work and the next generation. One can imagine a craftsperson 
building a boat or wood strip canoe, by hand. The work is comparable to manufactured efforts, but while similar, it 
does not regress to the level of detached reproduction by a human. The maker, the individual is engaged with the 
work. To some extent, the craftsperson is somewhat isolated from concerns of reality; in their own time, patient and 
still efficient, the work, not the schedule is of prime importance. In other words, the design is done when it’s done. 

Our vision of instructional craft includes a high level of implicit knowledge developed from experience. 
Craftspersons seek quality in both technical and aesthetic terms. They value the final product equally to, if not more 
than, the user or client; we expect physical manifestations of their work in their lives; calluses and patience. 

Many practitioners today might adopt this characterization for final phase of their work as an easy change 
in their current process of instructional design. Adding the title "craftsperson" to the completion phase of the work 
does not, however change the process. If the craftsperson is an appendage, a renaming of the manufacturing role, 
there will be no true design improvement, and will have the same real impact as calling in the graphic artist to apply 
tertiary visual and behavioral aesthetics at the completion of the project. For there to be value in craft, it must have a 
voice throughout the design, and exhibit real value in execution. Many in the field may share the values of the 
craftsperson, but still be constrained by practice, economics, or choice. The goal is to encourage thoughtful and 
engaged completion of instructional design projects. 

The building craftsperson, the mason of the Renaissance, evolved through time to become the 
contemporary architect (as buildings are now designed). There are still masons today, focusing more on production, 
but as design has become more complex and multifaceted, and as design has separated from construction, the role of 
architect has evolved as separate. As a profession architecture still values craft and seeks to train new architects in 
production techniques. It also educates practitioners in the results of research and values aesthetics and craft. This 
holistic view, a broader approach and perspective to the design process also applies to instructional design. 

Today, architects often integrate all the functions of the design process, from initial conceptualization to 
final evaluation. Architects are current with contemporary research and technologies, have the skills to work in 
various media, and integrate the needs of the user and the client through design ideas. 

The Instructional Architect (holistic conceptualization) 

We view the role of instructional architect as one who has a balanced approach to instruction design, one 
who values aesthetics and innovation, applies current research, and who critically examines solutions to increase 
user engagement, motivation, and interaction. Instructional architects are not satisfied by simply solving the 
problem; the architect is motivated by extending the boundaries of the resources to explore solutions that enhance 
learner experience, moving beyond the educational and technological specifications of the instructional problem (i.e. 
design beyond done), and ultimately striving for innovative and transformative potential. 

Developing an understanding of the entire project (i.e. having a holistic view of the design challenge) is 
critical to the design process. Inherent in this understanding is an identification and recognition of the assumptions 
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of the design problem, and a questioning of the design problem itself; what is the true nature of this design problem? 
This phase also examines the resources at hand and the theoretical and philosophical orientation of the project. The 
architect’s approach to instructional design attempts to balance utility, usability, and aesthetics (cf. Kirschner et ah, 
2004). By extending the engineer’s functional and usable solution and attempting to incorporate aesthetics at the 
core of the design process, the architect explores divergent solutions that extend and cultivate the affordances of a 
medium. 

Having a broad design perspective is essential to a project's success, but beyond that wide view, the 
designer needs to specifically address the development of new ideas. Unfortunately, in many design projects, a 
single driving concept is selected very early in the process and essentially “passed down through the ranks.”. These 
ideas are generally pre-conceptions, ideas of what works and what could easily be done, and sadly, they are also 
ideas that have already been previously executed. In order for innovative ideas to be adopted or even conceived, the 
successful designer needs to explore many alternatives; ideas that are different, unusual, that may fail or that will 
break the mold. In short, the instructional designer must also work as an artist. 
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Figure 2: Improved practice in instructional design. The artist provides divergent thinking, the architect provides 
holistic project understanding, but the project is still completed by a manufacturer. 

The Instructional Artist (playful experimentation) 

The instructional artist is an iconoclast; one who diverges from the norm and embraces experimentation 
and failure. Here the process of instructional design examines ideas that may not ultimately work, paths that are not 
expected, and allows for more diverse conceptualization. Within the field of creativity training, there are several 
techniques that encourage examining wrong answers or the opposites of the expected results. Similarly, the phase of 
the instructional artist is fraught with failure, and one that diversifies thought. Here is where most innovation in the 
field will occur. The wager of the artist is to win big, balancing attendant risk with the potential for substantial 
increases in the value of designs. 

We view artists as those with a mastery of a medium, with an intense focus on their work and a concern for user 
experience and aesthetics. They exhibit a high level of creativity, even to the point of working outside of society. 
Failure, unexpected results, and disturbance of the status quo mark the work of the artist; producing a finished 
product is not necessarily important, but rather the goal is to advance the understanding and development of new 
ideas. The artist stimulates divergent thinking at the beginning of any project; provides aesthetic direction and 
inspiration throughout the project, and acts as the "what if" person on a project team. 
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Figure 3: Role Based Design. The craftsperson advances the design to the next level; improvement and innovation 
is inherent and valued at each phase of the work. 



The artist is an instructional explorer. The artist uses instructional problems as stimuli to experiment with 
media and affordances. The instructional artist may work without client or audience, only later attempting to apply 
to instructional practice what has been learned through the artistic experience. The artist embraces failure and 
engages in continuous self-criticism while attempting to understand both the problem and self. 

Roles as process 

It must be understood that each aspect, each exemplar role participates throughout the design process. For 
example, parallel to the engineer perspective, the sensibilities of the artist must remain present. Complementary to 
the architect’s holistic perspective is the evolved and orchestrated execution of the craftsperson. We view these 
roles as generally sequential. Each role, in turn, leads the project, applying their own expertise: Artist, Architect, 
Engineer, and Craftsperson. This sequence is, of course, comparable in action to many other iterations of the design 
process, but given the use of these roles, these exemplars, each phase has its own values and quality. 

Each role, from the creativity of the artist, to the care and completion of the craftsperson is critical at some 
point in the process; each serves as check and balance for the other roles, the engineer bringing the artist back to 
earth, the architect reminding the craftsperson of the needs of the client; and each is constant and integrated into the 
entire process, not taking the lead all the time, but present and engaged throughout. 

As with many other disciplines, the methods and products of instructional design represent the values of the 
designer; our arguments here may be ones of belief or personal philosophy. Present today in the instructional field is 
a belief that design is a purely rational and logical solution of problems, or a belief that inherent in any design must 
be aesthetic, spiritual, and philosophical aspects, or a belief that design must be inclusive, and spring from the ideas 
and actions of the learners. The proposed Role Based Design perspective can be of value for instructional designers 
of all levels. Eor the experienced designer, a procedure for design is often already in place. The use of a role based 
perspective will remind the experienced designer of other, divergent aspects of design methods, and serve to 
stimulate directed reflection as part of the process. The Role Based Design perspective can also be used to organize 
and manage large teams or it can be used for projects designed by small teams or individuals. Applying new models 
or roles will help change the outlook and results. The attributes that follow are not to be used as a checklist, but 
rather as an understanding of different components of a complete designing experience. 

Eor the beginner, using Roles as a formalized linear design perspective, maybe even a process, can lead one 
through a challenging sequence of procedures; as artist, architect, engineer, and craftsperson. Using Role Based 
Design to complement ADDIE may encourage an inexperienced designer to include aesthetic components 
throughout the design process, to view the entire process as a whole, and be encouraged to innovate as opposed to 
replicate former design models. 
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Embracing Role-Based Design 

In lieu of a re-articulated summary of the Artist, Architect, Engineer, and Craftsperson roles, we conclude 
with a collection of 12 questions one can reflect upon before, during, and after each design project. The premise of 
Role-Based Design is illustrated in people, not steps or processes; Role-Based Design encompasses the values, 
mindsets, philosophies, characteristics, responsibilities, traditions, and practices of real designers. These are the 
attributes that ultimately lead to innovation. 

Artist (playful experimentation) 

• When listening to the initial problem, how did I freely explore a variety of aesthetic, technological, and 
pedagogical possibilities (rather than applying past design solutions to the current obstacle)? 

• What are some of the creative, unique, simplistic, complex, innovative, and bizarre ideas I exhausted when 
exploring the problem? 

• In what ways have I successfully failed during my design experimentation? 

Architect (holistic conceptualization) 

• What are the pedagogical, technological, and aesthetic characteristics/affordances of the proposed solution? 

• How does the conceptualized solution provide opportunities for transformation in learning and/or 
instruction? 

• What steps have I taken to create an instructional experience for the learner, as opposed to an instructional 
product? 

Engineer (scientific realization) 

• What are the physical, logical, pedagogical, technological, and cultural constraints of the design and 
implementation? 

• What structural and technical features have I implemented to ensure scalability and sustainability of the 
solution over time? 

• What measures have I taken to ensure a reliable, valid, and pedagogically-sound solution? 

Craftsperson (experienced evolution) 

• How have I improved upon the design conceptualized by the architect? 

• How have I affected the quality of ideas, processes, and production? What are 6 things I could have done 
better during this project (i.e. conceptual items, procedural items, and developmental items)? 

• What have I learned from this project that will ensure a higher quality of design and user experience for my 
next project? 
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Short Description 

The aim of this study is to explore whether presentation format and prior knowledge affect the effectiveness of 
worked examples. The experiment was conducted through a specially designed online instrument. A 2X4 factorial 
before-and-after design was conducted. Two-way ANCOVA and dependent t test were employed for data analysis. 
The study provided some direction for improving the quality of initial instruments for further studies. 

Background 

Well-designed instruction is an essential element in helping learners and teachers transfer knowledge 
effectively. To construct high-quality instruction, researchers suggest that the limitations of the human memory 
system need to be considered. According to the information processing theory, short-term memory has limited 
capacity (Atkinson & Shiffrin, 1968). Ignoring its limit may result in overloading the working memory and thus 
hinder learning. A possible solution to reduce the burden on working memory is to control extraneous cognitive load 
by using worked examples (Sweller, & Cooper, 1985; Sweller, Merrienboer, & Paas, 1998). 

A worked example usually consists of a problem statement, steps leading to a solution and a final answer 
(Renkl, 1997). Students can follow worked examples, provided step-by-step, to complete the target tasks 
smoothly. Although effectiveness of worked examples has been confirmed by many studies (Carroll, 1994; Zhu & 
Simon, 1987), there are still some areas of uncertainty. Some researchers have argued that worked examples are not 
universally effective for learners (Catrambone & Holyoak, 1989; Renkl, 1999; Ward & Sweller, 1990). The reason 
for the lack of efficacy in some circumstances may relate to the presentation format of worked examples, as well as 
learners’ prior knowledge levels. 

Some researchers have mentioned the difficulty of the inclusion and arrangement of text and diagrams 
(Ward & Sweller, 1990). Inappropriate arrangement of text and diagrams may distract the students’ attention and 
impose a higher cognitive load. Therefore, how to arrange text and diagrams in the design of worked examples 
requires more attention. Additionally, there is another factor that may affect the worked example design: the level of 
prior knowledge. In Kalyuga, Chandler and Sweller’ s (1998) study, they found that novices who have low prior 
knowledge benefited more from worked examples in text and diagram versions, but experts seem not to need the 
detailed information that text provides. As expertise increases, material that is essential for novices may become 
redundant for experts, imposing a high extraneous load. Therefore, to identify suitable worked example designs, 
presentation format and prior knowledge are essential variables that need to be considered. The aim of this study 
was to explore whether presentation format and prior knowledge affect the effectiveness of worked examples. 

Methods 

This study adopted a quantitative methodology that was supported by additional qualitative data. The 
experiment was conducted through a specially designed online instrument, and investigated whether presentation 
format and prior knowledge affect the efficacy of worked examples. A 2X4 factorial before and after design was 
used, including 2 levels of prior knowledge factor (i.e. low prior knowledge and high prior knowledge) and 4 
treatment conditions (i.e. no worked example, text-only, diagram-only and text-plus-diagram worked example). The 
participants in this pilot study were 1 senior undergrad student and 15 graduate students who major in computer 
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education related fields. In the beginning of the experiment, a 10-question pre-test about the IP address conversion 
between decimal and binary formats was used to determine the participants’ prior knowledge level. After it, the 
system randomly assigned the participants to different treatments. After reviewing the instructions, participants 
completed a 10-question post-test and a 9-point cognitive load scale. Score on the post-test, cognitive load scale, and 
time on the post-test session were used to measure the effectiveness of worked examples. A concept of instructional 
efficiency introduced by Paas and van Merrienboer in 1993 was also used to measure the effectiveness. For data 
analysis, two-way ANCOVA was employed to compare the mean difference in each group. In addition, dependent t 
test was used to compare the mean difference between scores on the pre-test and post-test, cognitive load scales, 
time spent on the tests and instructional efficiency. 



Results 

There was not a statistically significant result in the two-way ANCOVA and dependent t test in this study. 
The lack of significance may indicate that the knowledge and presentation format do not have an impact on the 
effectiveness of worked example or there are some other possible factors existed, such as individual visual 
preference, or that the sample size was too small to produce results. However, it may also indicate that the content of 
initial instruments may need some revisions. Although the reliabilities of the pre-test, posttest, pre-cognitive load 
scale and post— cognitive load scale are good, .68, .85, .98 and .99, it is better to do more detailed item analysis to 
improve the quality of instruments for the further study. Osterlind (1998) recommends that p values can be used for 
item analysis. Therefore, this study conducted item difficulty and item discrimination analysis on the pre-test and 
post-test. According to the analysis of the pre-test, some of the question options may need to be revised, because 
some of the options may be too difficult for participants and some of the options do not have strong item 
discrimination. On the posttest, there are some of questions also needed revisions. In addition, item analysis is also 
conducted to analyze the items in the pre- and post-cognitive load scales. The result showed that the items in pre- 
and post-cognitive load scale have high correlation with each other in each scale. Therefore, there are no revisions in 
the pre- and post-cognitive load scale. 

Except for the quantitative data, the study also received some comments from the participants. These 
comments leads to the following modifications: l.On the no-example treatment, a certain message should be 
provided to let the participants know they are in the control group and do not have any instruction in this session, in 
case they do not know what is going on in the treatment session, 2. Text-only worked examples needs to provide 
clearer explanation or description for participants, 3. Diagram-only worked examples may be too abstract for 
participant in low prior knowledge level group. 4. In the invitation letter or the beginning of the experiment should 
mention that participants are allowed to use paper and pencil to do the calculation. 

Conclusion 

From the above discussions, the human memory system has limits. To overcome these limits and help 
learners achieve schema automation, an appropriate instructional design is needed. Cognitive load theory suggests 
worked examples are a useful instructional design. Although the effectiveness of worked examples has been widely 
confirmed, attention needs to be paid to the presentation format and take individual difference into consideration. 
Therefore, the aim of this study was to examine the effectiveness of worked examples by manipulating these two 
factors and generate a better design for learners. This study primarily uses quantitative methods and conducts an 
online experiment to investigate whether the presentation format and prior knowledge affect the effectiveness of 
worked examples. A 2X4 factorial before and after design was conducted, including 2 levels of prior knowledge 
factor (i.e. low prior knowledge and high prior knowledge) and 4 treatment conditions (i.e. no example, text-only, 
diagram-only and text-plus-diagram example). Two-way ANCOVA and dependent t test were employed for the data 
analysis. Although there were no significant results in this pilot study, it indeed pointed out some directions for 
improving the quality of initial instruments for further studies. 
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Introduction 

Scientific literacy is cited as a key goal of education and the report of the National Academy, National 
Science Education Standards (NSES, 1996), identifies natural hazards as a central topic of the content of the 
standards for Science in Personal and Social Perspectives. The topic of natural hazards offers opportunities for 
students to engage in processes of risk analysis as well as analysis of societal implications due to a natural hazard, 
thus allowing for a relatively seamless integration of scientific thinking into the everyday contexts of real life. An 
ideal learning environment for the goals to promote scientific reasoning is the one which is amenable to high 
student-teacher interaction and discussion of content and accessibility to information. However, the introduction of 
critical and scientific thinking skills is specially challenging in the context of traditional undergraduate education, 
with large classes, limited technology resources, and a predominantly lecture-based instructional method. 

Research indicates that student ability to engage in scientific thinking is enhanced by the use of authentic 

problems, by scaffolding (or supporting) their selection of appropriate analytical tools and processes, and by 
providing opportunities for evaluating student reasoning. Also, research suggests that ill-structured problems, which 
tend to be amorphous and open to multiple correct answers based on justification, are more likely to engage students 
in complex thinking (Paul, 1993; Perkins, 1996). Thus, we decided to create problems based on real data - not 
‘laundered’ to be sterile, and we guided the students through the pitfalls and successes of solving real problems with 
real data. In our research, we used current events and situations that students might encounter in the news and daily 
lives as a context for the activities — problems using these current events were authentically ill-structured, meaning 
that a solution required data retrieval, analysis, evaluation, and synthesis by students. In addition, we also provided 
scaffolds to guide students' selection of strategies for complex, scientific thinking (Linn, 2000). We provided both 
procedural scaffolds to support the sequence of problem solving activities and cognitive scaffolds, where we 
integrated questions to stimulate reflection and evaluation of statements (Davis & Linn, 2000). Eor example, these 
included questions such as: what evidence can you provide to support your position? What alternative data could 
you present to support your position? These scaffolds make explicit the types of thinking that scientists engage in 
and trigger similar thinking in students. We also designed the activities to encourage student participation in peer 
discussions and collaborative activities, to promote the intellectual benefits during the course of learning 
collaborations and collective knowledge -building experiences (Scardamalia & Bereiter, 1996; Schrage, 1990; 
Vygotsky, 1978; Wertsch, 1991). 

In this research, we investigated the use of group-based problem solving activities to foster critical and 
scientific thinking in the area of natural hazards in a large undergraduate course with approximately 150 enrolled 
students. Instead of engaging students as passive listeners to lectures, our attempt was to engage them in more active 
learning, where they participated in small group and large group activities, notwithstanding the physical limitations, 
large number of students, and students' varied background and interest in science. 

Based on the purpose of this general education course, critical and scientific thinking skills are defined as 
the ability to identify the problem at hand while attending to the related context, to support arguments with credible 
data, and to conclude the influence and impact of the problem at hand. 

Earth 101 



The course: EARTH 101 intro 

EARTH 101: Natural Disasters: Hollywood vs. reality is a general education course (satisfying the natural 
sciences requirement) in the Department of Earth Sciences offered each semester at The Pennsylvania State 
University. During the semester, students study the causes, effects, and public perception of several natural hazards 
and disasters. The course typically has about more than 150 students enrolled, most of whom are non-science majors. 
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The overall goal of the course is to enhance scientific reasoning and critical thinking in the area of natural hazards. 
More specifically, it hopes to engage students in processes of risk analysis as well as analysis of societal 
implications due to a natural hazard. Class meets every Tuesday and Thursday for 90 minutes each. 

The instructor 

The instructor has been teaching EARTH 101 for several years, during which time he has built up a library 
of projects related to the topics covered in the course. In order to keep the course fresh and interesting to students, he 
would like to add different projects to his library. When designing content for the course, an immediate concern for 
the instructor is “how to improve learning in large science classes.” Although he often incorporates group 
discussions and cooperative activities in his class, there are obstacles to the success of such techniques in a large and 
diverse class like this one. Additional concerns he has are for teaching students how to communicate effectively, 
capturing students’ attention and linking knowledge with their interests, and incorporating real life situations into the 
activities. 

The environment 

The classroom is rather large and information is projected from a computer to a large screen at the front of 
the room. Many students arrive late for class, even on days of class activities. The course has more students than can 
ideally be accommodated for the purposes of in-depth discussion, question-and-answer interactions, and activities 
which promote learning. 

The large class size is a constraint. It is difficult to manage the logistics of delivery of information and 
discussion in such a large class. Furthermore, the instructor often divides students into a number of groups in order 
to explore a complex issue. Due to the fixed chairs in the class, it is difficult to discuss a question given by the 
instructor for group members. Thus the physical constraints reduce student ability to engage in active and group 
based learning. In addition, the lack of technology mandates use of traditional materials for in-class work. 

The learner 

Most students chose to enroll in the course because it fulfilled a general education requirement for them. 
Many students come in late and some are engaged in activities not related to course content (napping, reading a 
novel, etc.). Moreover, few students asked questions during class. Learners are generally not science majors and they 
have low prior knowledge about the topic of natural hazards. This means they need to learn a great deal about 
scientific reasoning and the attributes and consequences of natural hazards. The learners have varied interests and 
not all of them are highly motivated. This means that the instructor will have to provide content that can be tied to 
many students’ lives and particular interests in an exciting fashion. We cannot say that all the students in this course 
have strong intrinsic motivation to learn this topic to begin with. Therefore, the instructor, content, and methods of 
delivering the content needed to be structured in a way that is extrinsically motivating, such that intrinsic motivation 
may increase. 



The design 

In our study, we designed instructional materials for 2 units (Hurricane Smith, Global Warming) for the 
course. One activity focused on hurricanes and their impact on society, while the other focused on global warming, 
with specific reference to its impact on Bangladesh. Each unit typically contained three discrete but interrelated parts, 
individual information packets, group worksheet, and final individual report (Figure 1), with each part building on 
the previous one. The learning unit starts with the individual factual worksheet, wherein students work through teh 
assigned information packets and individual worksheet as a preparation for their group work in the next class 
meeting. During the group activity, students were engaged in activity that required them to present, evaluate, and 
finally to come to an agreement or conclusion on the designated tasks. After the group work session, students wrote 
individual reports to describe their personal judgment on the topic/issue related to nature disasters covered in class. 
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Figure 1 . Geosciences Learning Unit Structure 




Thus, the design focused on using collaborative and active learning to guide student through the process of building 
up their critical thinking skills. The representation and analysis of group thinking will be the focus of this paper. 

In the first activity, we divided students into 18 groups of approximately 8 students each, and each group of 
students was assigned to a specific town that was in the path of an oncoming hurricane. Students were also assigned 
to specific administrative roles within the town/community such as being a member of the town council, the mayor's 
council, the health board, the tourist board, the disabilities advocates group, and other civic roles. Students were 
provided with group worksheets and as a team they had to evaluate the position of the hurricane and decide on an 
action (evacuate or not) based on the data provided to them as well as by sifting through the various arguments in 
favor or not of evacuation. The final product is their community and group worksheets which documented their 
reasoning and decision. 

In the second activity that focused on the impact of global warming in Bangladesh, students were divided 
into 36 groups with approximately 4 or 5 students in each group. Students first worked on an individual worksheet 
using various reference materials provided to them, and then convened as a group to discuss and create a list of the 
most important aspects of global warming's impact on Bangladesh society. In this activity, instead of proposing 
solutions (which have not yet been arrived at by a consensus in the scientific community), we specifically wanted to 
see students’ perspectives on the issue of Global Warming. Since the purpose of documentary is to present 
perspectives on issues, the final document for this activity was a storyboard that described a video documentary on 
Bangladesh and global warming. 

Both group activities have a list of prompting questions to serve as procedural and cognitive scaffolds to 
guide students through the thinking process. The elements of critical and scientific thinking skills are implicitly 
embedded into the activity. Thus, these two group activities both served as the instrument and assessment for 
students’ critical and scientific reasoning skills. 



Data Analysis 

Our focus in this study was to examine evidence of critical and scientific thinking as evidenced through 
group work. 

Two sets of analytic scoring rubrics (Nitko, 2006) were developed to assess students' work. The rubric for 
Hurricane Smith was adapted based on the Critical and Integrative Thinking (CIT) rubric developed by the Center 
for Teaching, Learning, & Technology at Washington State University. We assessed students thinking focusing on 
their abilities to (1) identify problem, question, or issue; (2) consider context and assumption; (3) present and 
analyze supporting data; (4) identify conclusions and implications, with each ability graded on a scale of 1-6 ranging 
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from emergent to mastery. However, later in the actual data analysis phase, we had to abandon the rubric we 
developed and to use another holistic rubric and checklist to assess students’ work. The reason of doing so will be 
discussed in later section. Based on students’ response in the group worksheet, the revised coding schemes for 
Hurricane Smith were formed according to the question type in the worksheet: (1) description; (2) explanation; (3) 
decision points; (4) n/a (blank), (see Table 1) 



Table 1 . Hurricane Smith coding scheme. 



Category 


Description of the category 


Example 


Description 


Plain facts 


category 4 

5-10 inches rainfall 

Flooding + debris from wind 


Explanation 


Give accounts to the why question 


storm surges cause flooding 

flooding + waves because of flooding to the 

coast->displaced citizens from home 


Decision points 


To evacuate or not 


do not evacuate 

keep an eye on data 

tell citizens to prepare for evacuation 

evacuate 


n/a 


No answer, blank 


No answer, blank 



For the Global Warming activity, we used holistic rubrics and checklists to analyze group worksheets. All 
group worksheets were first categorized into three broad categories: descriptive, explanatory, predictive. The three 
categories were further refined as other themes began to emerge. At the end, a complete coding rubric for Global 
Warming activity contains four main categories and six subcategories: (1) descriptive (fact, storyline, data); (2) 
explanatory (explanation, intent, solution); (3) predictive; (4) n/a (blank) (see Table 2). 



Table 2. Global Warming coding scheme. 



Category 


Description of the category 


Example 


Descriptive 


Fact 


Presents facts, issues relating to the 
topic 


Normal everyday life: facts and info about 
crops, especially rice, fishing 


Storyline 


Storyboard contains dramatic storyline 
such as family reunion, city rebuilding 


Hut destroyed: As he approaches the house, 
he sees it is destroyed. His first thought is, 
“Where is my family?” 


Data 


Use of specific data or numbers in the 
storyboard 


Bangladesh density: 138,448,210 people 
June- Sep: Bulk of rainfall for year, 80% of 
Monsoon rainfall 


Explanatory 


Explanation 


Provide explanation, reasoning for 
aspects mentioned in the storyboard 


Potential global warming ocean rise: ice caps 
are melting causing water level to rise 


Intent to explain 


Exhibits intention to provide 
explanation, not directly identify the 
causal relationships 


Pre-flood: Handful of farmers talking about 
how the pre-monsoon season hasn’t brought 
any rain. 


Solution 


Provide solution to the situation at hand 


What people can do: what people can do to 
lesson impact on global change, people on the 
street, scientists. Things we can do to help 


Predictive 


Prediction 


Provide prediction as to what will 
happen in the future 


Conclusion: Tell what would happen in the 
future (50 years, 100 years, 1000 years) 




n/a 


No answer, blank 


No answer, blank 
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Seven community worksheets and a total of 46 group worksheets from Hurricane Smith activity were 
collected for data analysis. For Global Warming activity, 35 storyboards entered the data analysis phase. One 
storyboard was excluded because it employed only graphical presentations of ideas and was therefore not suitable 
for textual analysis. Coding of the two group worksheets was conducted by two raters. For Hurricane Smith, 
reliability was assessed by having 30% of the group worksheets coded by the two raters. Agreement on coding for 
Hurricane Smith reached 87.5%. For Global Warming, reliability was assessed by having 20 % of the storyboards 
coded by the two raters, and the agreement reached 87.7%. 

Results and Discussion 



Hurricane Smith 

Overall, the data shows that students did not answer the prompting questions as evidenced in the worksheet 
completion types and rates. In terms of community and group decision points, although we specifically asked the 
students to note down data source they used to support their decisions, in the community worksheets 28.6% of the 
total data source box did not provide data source, and 43% of the data source box was left blank, whereas in group 
worksheets 26% of the total data source box did not provide data source and 5% of the data source box was left 
blank. As for the open-ended questions in community worksheets, regardless of the “why” question asking for 
justification and explanation, only 31% of the total “why” question in community worksheets has the explanation 
element whereas the percentage in group worksheet was 57% (of the total “why” question). A general trend in the 
completion rates of Hurricane Smith activity worksheet was that students were more willing to complete group 
worksheets than community worksheets. 



Figure 2. Community and Group decision points 



Community Decision Points 




Types 




* w/src; support with source, w/o src; without source to support, n/a: blank 



Figure 3. Community worksheet open-ended questions 
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Community Open-Ended Questions 



T oken 




□ Description 

□ Explanation 



T ypes 



* total d= 42; total e=49; d: description, e; explanation, n/a: blank 



Figure 4. Group worksheets open-ended questions 



Group Open-Ended Questions 



Token 




□ Description 

□ Explanation 



Types 



* total d= 42; total e=49; d: description, e; explanation, n/a: blank 

A closer look into the open-ended responses that students produced in community and group worksheets 
revealed that students tended to write very short answers. Often times, only short phrases or even single words were 
provided. This lack of detail was the major reason why we abandoned our original rating rubric developed from the 
Critical and Integrative Thinking (CIT) rubric and used other rating scale instead, since the students’ group 
worksheets did not exhibit enough information to meet the level of thinking we previously defined in the rubric. 



Table 3. Sample student response 



Group designation 


Info release time 


Example 


Deerfield A, mayor/city 
council 


F‘ time release 


Strong winds — dangerous because of trees and houses 
Flooding — people can’t evacuate 


Deerfield A, infrastructure 
services 


3“* time release 


We made the correct decision to evacuate westward 
rather than to the north 


Deerfield A 


F‘ time release 


Expected path goes right over us 
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Evidence of reasoning for the “why” question was very little if there was any. As mentioned earlier, 
students tended to give only single word or short phrase answers for both descriptive and explanation questions, and 
it was thus difficult to infer the reasoning behind students’ thinking from the short answers they gave. At times, 
longer pieces of response were also found in the group worksheet. 



Table 4. Sample student response to “why” question 



Community, Group 
designation 


Info release time 


Example 


Deerfield A, disability 
services 


1 time release 


Elderly & medical care 


Deerfield D, emergency 
services 


3“* time release 


Winds, flooding, poor travelling condition 


Deerfield D, disability 
services 


1 time release 


The winds are so high that it might cause the power to 
fail. Without power, those on life support and machine 
will potentially die. 



At this point, we need to ask ourselves what did all these limited response in the community and group 
worksheets imply? The worksheet apparently did not serve the function of documenting students’ reasoning to make 
their thinking visible as we originally thought they would be. Several factors might come into play to result in this 
type of limited response, namely, (1) time constraint, (2) extrinsic motivation constraint, and (3) design constraint. 

To begin with, students were given approximately 40 minutes for their group work. Considering this time period and 
the task students need to complete, they were actually working under a considerable time constraint. In this activity, 
students need to first discuss questions within their assigned roles as a group and then convene to discuss the 
questions as a community. There were a total of 33 questions (see Table 5 for sample questions) on the group 
worksheet for students to complete, and although they are repetitive prompting questions (because of the three 
information release points), the time was not sufficient for completing all those questions. As a result, students 
produced very short answers in order to finish the task in time. 



Table 5. Sample prompting questions 



1 


List below the dangers and impacts associated with Hurricane Smith in its current location. 


2 


What do you think are the three most important dangers/impacts to address from the list above? Why are 
they important? 


3 


What is your decision? 


3.1 


Why did you make that decision? List your reasons and the data that you used as evidence. 



Another thing immediately related to the time constraint was the single word/phrases we saw in students’ 
group worksheets. It might be that given such a short amount of time, students considered the time might be better 
spent on discussion rather than writing. The group worksheets did provide good guiding questions for their group 
discussion. One of our colleagues who went to the classroom for observation confirmed with us that this activity did 
generate enthusiastic discussion. However, that kind of discussion was not documented on group worksheets. There 
is also the extrinsic motivation constraint that could have affected students’ willingness to spend time on the writing 
task. Students received only participation scores (i.e., check minus, check, check plus) for in-class activity. The fact 
that the in-class activity does not count in their final semester grades could have influenced students’ motivation for 
this group work. The last factor that might have resulted in students’ limited response came from the design of the 
group activity itself. There were three information release points in the Hurricane Smith activity where students need 
to make decisions based on the different information (the status of the Hurricane) they were given. However, the 
decision making process was essentially the same each time (identify problem in their context, support their 
argument, conclude their decision), and thus the prompting questions became repetitive such that it is likely that 
most students perceived the prompting questions as a discussion aid rather than a place to document their thinking. 
The above mentioned constraints turned out to be crucial for consideration while addressing future modification of 
the group work if our purpose is to document and make students’ thinking visible. 
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Global Warming 



From the analysis for group activity in global warming, we know that students concentrated mainly on 
providing facts or using stories in expressing their thinking. Only 22 percent storyboards are presented with actual 
explanations of scientific data pertaining to societal impacts or contain an intent to explain such connections (see 
Table 6). This low percentage of explanation may be contributed to the prompting question sheet we gave students 
before they engage in group storyboard activity. Although scaffolding questions were also presented for students 
before they engaged in the group storyboard worksheet activity, the prompts do not specifically ask students to put 
down justifications on their storyboards. As a result, while they selected one or some impacts of global warming on 
Bangladesh in their storyboards, they did not express their explanation in the storyboards despite the scaffolding 
questions. Another factor that might lead to this high percentage of descriptive and low percentage of explanation 
could be the one example of the sample storyboard that was provided to the students in the scaffolding question 
sheet. The sample storyboard was descriptive in nature; students might be following that sample storyboard when 
they engaged in this group activity. Moreover, only 3 percent storyboards included a prediction of what would 
happen in the future. These low percentages should not be considered as drawbacks since the overall objective of the 
activity is to see students’ different perspectives on Global Warming and this activity did not specifically ask for any 
type of content in the storyboards. However, the fact that we did not specifically ask for a certain type of content in 
the storyboard is also a double edge-sword in that we were unable to see students’ justification for their choice. 
Should this type of activity format to be used in the future, one important modification is to ask for students’ 
justification wherever and whenever suitable. 



Table 6. Global Warming storyboards analysis 



Category 


Descriptive (72.4%) 


Explanatory (22.2%) 


Predictive 


n/a 


total box 


code 


a 


b 


c 


d 


e 


f 


g 


n/a 




token 


226 


41 


7 


14 


51 


19 


13 


7 


378 




59.79% 


10.85% 


1.85% 


3.70% 


13.49% 


5.03% 


3.44% 


1.85% 





* a: fact, b: storyline, c: data, d: explanation, e: intent to explain, f: solution, g: prediction, n/a: blank 



Figure 5. Types of storyboards by category 



Global Warming Storyboards 




* a: fact, b: storyline, c: data, d; explanation, e: intent to explain, f; solution, g: prediction, n/a; blank 

One thing that we encouraged students to do was to use their creativity to approach this documentary 
storyboard activity. As a result, there was one group that used purely drawings to represent their thoughts. However, 
this pictorial representation of ideas made it difficult for us to infer the thinking behind their drawing. Another type 
of storyboard that might have been influenced by this call to be creative was the Hollywood like storyline storyboard. 
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This specific Hollywood like storyline presents a dramatic story of a family household being separated by flooding, 
where the father goes on a journey in search of his family, and finally the family is reunited (as can be seen in Table 
2). One might suspect that the use of movie clips in this course influenced the purpose of this activity. However, as a 
matter of fact, not all movie clips shown in the class belongs to the Hollywood genre, nearly half of the movie clips 
also belong to the documentary genre. Although the scaffolding questions directed students’ attention to the potential 
impact of global warming in Bangladesh, some groups of students produced this type of storyboard that did not 
exhibit perspectives that were directly related to the scientific aspects being explored. 

The focus of this documentary storyboard activity is for students to demonstrate their perspectives on the 
impact of global warming in Bangladesh. A question that needs to be answered is that whether this design provided 
students the opportunity to consider the various impacts on the society. The analysis results at a first glance might be 
disappointing with the majority of the storyboards being descriptive in nature. However, although descriptive, we 
can also see students trying to approach the issue from different angles, for example, the dense population in 
Bangladesh, its geographic location, and its economy. It would be unfair to say that there was little reasoning behind 
this descriptive presentation of different phases of the Bangladesh society. However, what we can infer is that the 
format did not provide a space for students to demonstrate their reasoning. Since little written evidence of students’ 
thinking was explicitly presented in their storyboard, the question of whether our design affords students to think 
critically can only be answered from their follow up individual report. 

Conclusion 

In this study, we investigated the use of group-based problem solving activities to foster critical and 
scientific thinking in the area of natural hazards in a large undergraduate course. Instead of engaging students as 
passive listeners to lectures, we attempted to engage them in more active learning, where they participated in small 
group and large group activities, regardless of the physical limitations, large number of students, and students' varied 
background and interest in science. The representation and analysis of group thinking as documented in students’ 
group worksheet was the focus of this paper. However, little trace of higher level thinking was found from the 
analysis of the group worksheets. The majority of the group worksheets only had limited response that were short 
and descriptive in nature which prevent us from inferring any complex thinking behind those limited responses. 
Several constraints such as time, motivation, and design might be the causes that contribute to students’ limited 
responses. 

Limitations 

All too often, designers picture a perfect design that is seldom realized in reality. We started off this design 
with the assumption that students would follow and complete every step in the group worksheet. However, we failed 
to take other constraints into consideration while doing the design, which ultimately resulted in students’ limited 
responses in their group worksheets. Therefore, we would have to acknowledge that our design did not successfully 
made students thinking visible allowing for analysis and assessment of their critical thinking skills. We learned that 
if the purpose is to document students thinking process, the time, student motivation (graded or not), scaffolding 
questions, and the final product that we asked of students needed to work together to achieve the purpose. 

In terms of the time constraint, a balance needs to be reached between the students’ task workload and the 
class time they have for the group activity. This constraint is more prominent in Hurricane Smith activity than in 
Global Warming activity since students were not required to justify their decisions in their storyboards. If balance 
was not reached, no matter how thorough the scaffolding questions are, they are at best a list of discussion guides 
that students touched on rather than being written down. Time is better spent on discussing rather than writing as 
evidenced in the completion rate of the group worksheet; the completion rate of Global Warming storyboard is much 
higher than that in Hurricane Smith. 

Student motivation certainly has its role to play here. Some extra “external motivators’’ such as grading 
needs to be added if we want the students to take these activities more seriously in terms of completion and the 
quality of the final product. The fact that the in-class activity does not count in their final semester grades could 
perhaps have exerted some influence on students’ attitude toward group work. Students treat the completion of 
quality of the final product lightly because they know that the efforts they put in do not pay off. Letting students 
know that their efforts in this class do pay off at the end of the semester is always a good strategy to keep students 
focus on the tasks assigned. 
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One thing that should also be mentioned is the form of final product that we required from the students. If 
we specifically want to see students’ reasoning in the document, we will need to put special emphasis on that point 
and clearly convey it to students in both oral and written instruction for the activity. Such reminders could also serve 
as a meta-cognitive scaffold to focus students’ attention on their own thinking process. 

Future Directions 

Although this study did not have the written evidence to claim that our design helped to facilitate students’ 
critical and scientific thinking (the effectiveness of this design), in-class observation conducted by one of our 
colleagues showed that students were actively engaged in these group activities and that their in-class discussion 
revealed this higher level thinking we sought to promote. In the end, perhaps a better way to approach this type of 
learning is to use the group activity only as an instrument to facilitate students’ critical and scientific thinking instead 
of an assessment of students’ critical and scientific thinking due to practical constraints such as time, class grading 
policy, and format (hand-written or not). Or, other more effective ways of documenting students’ thinking in their 
in-class group activity (such as recording) will need to be implemented to achieve this purpose. Our next step will be 
to modify our design and revise the activity according to the lesson we have learned from this current 
implementation to see whether students’ reasoning skills can be demonstrated and documented in their group 
worksheets. 



References 

Davis, E., & Linn, M. C. (2000). Scaffolding students’ knowledge integration through prompts for reflection. 

International Journal of Science Education, 22(8), 819-837. 

Paul, R. (1993). Critical thinking: How to prepare students for a rapidly changing world. Santa Rosa, CA; 
Foundation for Critical Thinking. 

Perkins, D. N. (1996). Foreword. In B. G. Wilson (Ed.), Constructivist learning environments: Case studies in 
instructional design (pp. v-viii). Englewood Cliffs, NJ: Educational Technology Publications. 

Linn, M. C. (2000). Designing the knowledge integration environment. International Journal of Science Education, 
22(8), 781-796. 

National Research Council. (1996). The National Science Education Standards. Washington, DC; National Academy 
Press. 

Nitko, A. J., & Brookhart, S. M. (2006). Educational assessment of students. New Jersey: Pearson. 

Scardamalia, M., & Bereiter, C. (1996). Adaptation and understanding: A case for new cultures of schooling. In S. 
Vosniadou, E. D. Corte, R. Glaser & H. Mandl (Eds.), International perspectives on the design of 
technology-supported learning environments (pp. 149-163). Mahwah, NJ: Lawrence Erlbaum. 

Schrage, M. (1990). Shared minds: The new technologies of collaboration. New York: Random House. 

University, Washington State University (n.d.). Guide to rating critical and integrative thinking. Retrieved from 
Washington State University Critical Thinking: http://wsuctproject.wsu.edu/ctr.htm 
Vygotsky, L. S. (1978). Mind society: The development of higher psychological processes. Cmbridge, MA: Harvard 
University Press. 

Wertsch, J. V. (1991). Voices of the mind; A sociocultural approach to mediated action. Cambridge, MA; Harvard 
University Press. 



133 




Implementation of Web-Enhanced Features for Successful Teaching and 
Learning. The Utility of Technological Progressions of Learning Management 

Systems 



Dirk Ifenthaler 

Albert-Ludwigs-University Freiburg, Germany 

Abstract 

Learning management systems (LMS) integrate interactive learning environments and administration and enable 
customization of online instructional materials. However, we argue that the use of LMS’ s is predominantly content 
oriented and that they fail to promote meaningful learning environments. Our three quantitative and qualitative 
research studies helped us to investigate our concerns about the use of LMS features. One study focused on 133 
blended learning courses implemented in Moodle, with a total of 3643 students enrolled. Our second study analyzed 
18 courses from a bachelor program over a three year time span. Additionally, we conducted expert interviews in 
order to identify a practicable taxonomy for designing effective online instruction within the LMS Moodle. The 
findings indicate that most instructors use LMS’s for sharing documents and do not make use of their technological 
and instructional advantages. With this instructional regression in mind, we discuss the necessity of a taxonomy of 
LMS features for teaching and learning. 



Introduction 

The rapid progress of computer technology has ushered in a new culture of innovative teaching and 
learning in schools, higher education, and lifelong learning. Multimedia technologies implemented in computer- 
based learning environments provide new opportunities to enhance traditionally taught courses (Alessi & Trollip, 
2000; Hakkinen, 2002; Land & Hannafin, 1996; Mayer, 2001). Web-enhanced features (e.g. synchronous and 
asynchronous communication, document sharing, etc.) in particular grant access to education at any time or place. 
Additionally, the omnipresence of the internet and high-bandwidth connections has brought about the development 
of powerful software packages called learning management systems (LMS). An LMS integrates the administration 
and facilitation of online activities and the distribution of learning materials (Falvo & Johnson, 2007). 

However, in this paper it is argued that the use of LMS’s is limited to only a few technological modules and 
that they fails to promote meaningful learning environments. Although LMS’s have the technological and 
instructional potential to support a wide range of learning activities, such as exploring, constructing, and 
manipulating models, solving authentic world problems, or articulating and discussing individual ideas, they are 
simply used for sharing documents. Therefore, we conducted three empirical studies to investigate our concerns. 

Our first research interest is how instructors design their online courses and which LMS features are implemented. 
Second, we investigate the student’s usage of the LMS. Third, we want to identify a practicable taxonomy for 
designing effective online instruction within the LMS Moodle. 

Learning Management Systems 

Learning management systems (LMS) integrate interactive learning environments as well as course and 
user administration (Black, Beck, Dawson, Jinks, & DiPietro, 2007; Waterhouse, 2005) and facilitate customization 
of online instructional materials (Koszalka & Ganesan, 2004; Schulmeister, 2003). Baumgartner, Hafele, and Maier- 
Hafele (2002) report in their analysis more than 120 different commercial and open source LMS products (e.g. 
Blackboard, eCollege, Moodle, WebCT). Another study by Falvo and Johnson (2007) identified the most popular 
LMS’s used at colleges and universities in the United States. A random sample of 100 institutions revealed the most 
frequently used LMS’s to be Blackboard (www.blackboard.com) and WebCT (www.webct.com). However, more 
research is needed to investigate many critical questions about online instruction (Falvo & Johnson, 2007). 
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Similarly to a content management system (CMS), an LMS consists of a database where various types of 
information are stored. Figure 1 shows the comprehensive characteristics of a LMS. The management and 
administration of courses, authors, learners, and instructors involves selecting between several possibilities for 
circulating important information, changing access permissions, and granting privileges to use to certain functions of 
the LMS. Authoring tools help instructors and course creators in developing courses, content, and assignments. 
Numerous evaluation and diagnostic features help to asses and analyze the student’s knowledge, provide examples 
for feedback, and help instructors with grading of students’ assignments. Additionally, communication via chat, 
forums, and email connects all users of an LMS (e.g. instructors and students; authors and instructors; authors and 
administrators, etc). 



1 

Course Organization / Your Understanding of Research 

Q Course Syllabus 
Q Course Organization 
Q Howto create a concept map 


□ 


Q Assignment ffl 




Q Concept Map #1 




![^ Upload Assignment ft\ and Concept Map 




2 


□ 


The Process of Conducting Research 








Q Mitgram Experiment 




Q Process of Research 




Q 1993_Vooijs_TelevisionViolence.pdf 





Figure 2; Screenshot of two Moodle course units 

LMS’s were developed to support learning in several ways. Accordingly, the technological progressions of 
LMS offer new opportunities for course designers, instructors, and students. They provide learning materials, tasks, 
and exercises, offer synchronous and asynchronous communication tools, evaluate student activities, and support 
instructors with their course administration. However, the rapid introduction of LMS into almost every university 
and organization as a teaching, learning, and management tool (Baumgartner et ah, 2002; Bett & Wedekind, 2003; 
see Falvo & Johnson, 2007; Trahasch, Kraus, & Eferth, 2002) was not accompanied by a precise investigation of the 
instructional implications of these technological based systems (see Schulmeister, 2003). 
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For a further investigation of the available features of an LMS and their instructional potential, we need 
only to concentrate on one system. As the LMS Moodle (Modular Object-Oriented Dynamic Learning Environment) 
is a widely accepted and used open-source product (Melton, 2006) which has features similar to Blackboard or 
WebCT, we will investigate our research questions using Moodle (see Figure 2). 

Moodle provides various features for the design of online instruction (Cole, 2005; Hobarth, 2007; 

Williams, 2005). Table 1 shows the features of the Moodle Version 1.8.4 (www.moodle.org) we used for the studies 
reported in this paper. 



Table 1: Features available in the LMS Moodle (Williams, 2005) 



Feature 


Short Description 


Assignment 


The instructor can provide written feedback or grade assignments 
submitted online by the student. 


Chat 


The module allows a real-time synchronous discussion. 


Choice 


A question by the instructor with a choice of multiple responses. 


Files / 
Resources 


Uploaded files for download (e.g. text documents, spreadsheets, 
slides, sound, graphic, or video). 


Forum 


The module allows asynchronous discussions between students and 
the instructor. 


Glossary 


Allows one to create and maintain a list of definitions. 


Journal 


The module enables students to reflect on a particular topic. The 
entries can be edited and refined over time. 


Label 


The module enables the instructor to add text or instructions to the 
content area of the course. 


Lesson 


Content is delivered in an interesting and flexible way, including 
grading and questions. 


Quiz 


The module allows the instructor to design a set of short tests. 


Scorm 


Uploaded and implemented SCORM packages as part of the course. 


Survey 


Standard surveys for gathering data from students (e.g. ATTLS, 
Critical incidents, COLLES). 


Wiki 


Enables the authoring of documents collectively in a simple markup 
language. 


Workshop 


Students are enabled to assess each other’s projects in a number of 
ways. 



Research Questions and Hypotheses 

Learning management systems like Moodle enable instructors to easily create individual online courses. 
Moodle provides a variety of modules and resources for building comprehensive online instruction. For example, 
WYSIWYG editors and other tools help to customize the page layout, links, resources, assignments, forums, chats, 
and much more. New versions and updates of available LMS’s almost always include new features and possibilities 
for instructors. On the other hand, Schulmeister (2003) argues that the application of such LMS’s leads to a 
regression in the design of online learning and that many features are hardly used at all. Accordingly, our first 
research question is: 

(1) Do Moodle course instructors use all available activity modules and resources for building their online 
instruction? 
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Particularly in blended learning courses (Kerres & de Witt, 2003), the use of LMS’s can improve the 
chance of access to resources. Learners can access them independently without consulting the instructor at any time 
and from any place. As the particular requirements change in the course of a six semester bachelor program, we 
assume that this also has an effect on the use of the LMS. Accordingly, a second research question investigated in 
this article is: 

(2) Do the access frequency and activities of students change in the course of a six semester bachelor program? 

Although LMS’s like Moodle have the technological and instructional potential to support a wide range of 
learning activities, such as exploring, constructing and manipulating models, solving authentic world problems, or 
articulating and discussing individual ideas, most people simply use them to share documents. Therefore, it seems 
necessary to provide a taxonomy of the instructional value of the features of the LMS Moodle (see Koszalka & 
Ganesan, 2004). However, we argue that such a taxonomy must have a sound theoretical basis and should be 
empirically tested. Hilgenstock and Jirmann (2005) introduced seven instructional functions for online instruction, 
including information, communication, cooperation, assessment, self-reflexion, feedback, and evaluation. 
Accordingly, a third and last research question to be explored in this article is: 

(3) Does a pairwise comparison of LMS features and instructional functions conducted by Moodle experts help 
to identify a practicable taxonomy for designing effective online instruction? 

We investigated our three research questions in three different studies, which we will present in the 
following section. We used both quantitative and qualitative data analysis to acquire adequate information for each 
research question. 



Study 1 

The center of interest of our first study was on how instructors design their undergraduate and graduate 
online courses. The LMS the instructors used in this study was Moodle (Modular Object-Oriented Dynamic 
Learning Environment). All activity modules and resources of Moodle were available to the instructors. As most of 
the Moodle data is stored in an SQL (structured query language) database, we were able to assess the data and 
investigate which activity modules and resources were implemented by the instructors. Additionally, we were also 
able to study the students’ usage of the LMS. 

Method 



We gathered and assessed data from 133 courses of an undergraduate program in instructional design (n = 
73) and a graduate program in educational science (n = 60) at a German university, %^(1, N= 133) = 1.27, p> .05. A 
total of 3643 students were enrolled in these courses {M = 27.39, SD = 12.93). 

Procedure 

The research study focused on 133 blended learning courses (see Kerres & de Witt, 2003) that included 
various face-to-face and online activities available to students through the LMS Moodle (Modular Object-Oriented 
Dynamic Learning Environment). The course instructors could implement 14 different activity modules and 
resources (assignment, chut, choice, forum, glossary, journal, label, lesson, quiz, resources, SCORM, survey, wiki, 
and workshop', see above for a detailed description). 

All activities within the LMS Moodle are stored in several relations and tables. The Moodle database 
consists of 187 tables linked by unique identifiers. In order to access the necessary information, various queries to 
the database must be executed. The tables of the database can be assessed by the formal language SQL (Structured 
Query Language). However, the Moodle database is not well documented. Therefore, all tables and dependencies 
needed to be studied explicitly before we were able to retrieve the necessary information from the database. Eigure 3 
shows an example of an SQL statement. Here, forums implemented in the course with ID number 157 are being 
requested. 
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SELECT * 

FROM mdljog' 

WHERE course' =157 
AIID 'module' LIKE 'forum' 
LIMIT 0 , 30 



Figure 3: Example of an SQL statement 

Additionally, using the Moodle database we were able to assess the data on each individual course, 
including the number of enrolled students, hits (access and clicks per course and student), forum posts, and 
implemented activity modules and resources. Student usage of the implemented activity modules and resources was 
assessed using the log files stored in the Moodle database. Additionally, the log files provided information on the 
students’ hits as well as detailed information about individual usage of activity modules and resources. 

Results and Discussion 

Implemented Activities and Resources 

The average amount of modules implemented in the 133 Moodle courses was 34.61 {SD = 32.04). A total 
of 3145///ei were uploaded, which makes this module the one most frequently implemented (M = 23.65, SD = 
29.83), ranging from 0 to 176 in individual courses. The module forum was used a total of 999 times in the courses 
(M = 7.51, SD = 8.41, Min = 1, Max = 54). The other modules were implemented less than once per course on 
average, and the modules SCORM and workshop were not used at all (see Figure 4). 




Figure 4: Sum of implemented modules 



There were no significant differences between undergraduate {M = 37.62, SD = 29.46) and graduate 
courses {M = 30.95, SD = 34.82) with regard to the overall sum of implemented activities and resources, f(131) = 
1.17, p > .05. However, further investigations of the 14 different activities and resources revealed significant 
differences between the implementation of the activity module assignment in undergraduate (M = .60, SD = 1.58) 
and graduate courses (M= .05, SD = .39), f(131) = 2.65, p < .05, d = .47. 
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Moodle Usage 

Total use of the LMS Moodle ranged from 14 to 46,759 hits (M = 4897.95, SD = 7998.04). Undergraduate 
students used Moodle significantly more (M = 6686.86, SD = 9841.29) than did graduate students (M = 2721.45, SD 
= 518.78), f(131) = 2.93, p < .05, d = .57. The average amount of posts in the discussion modules was also 
significantly different between undergraduate (M = 79.92, SD = 163.19) and graduate students (M = 31.55, SD = 
75.41), f(131) = 2.12, p < .05, t/= .38. 

Discussion 

The findings of our first study have considerable significance. The most frequently implemented Moodle 
feature was the resource. Other features (e.g. SCORM, workshop) were not implemented at all. Aside from the 
features resource and forum, the available features were hardly implemented by the course instructors at all. 
Accordingly, we conclude that the LMS Moodle was used primarily to share documents. Such low use of the 
features of the LMS raises the question as to whether such a highly elaborated technical system is necessary at all. 

Study 2 

Our second study investigated students’ behavior in the LMS Moodle over a three year period. The three- 
year bachelor program in instructional design at a German university included a total of 15 required courses. 
Optional courses were not considered in this study. 

Method 

Subjects 



A total of 18 students (10 female and 8 male) were enrolled in the three -year the bachelor program in 
instructional design. Their mean age at the beginning of the course of study was 22.98 years (SD = 2.31). Since all 
of the students belonged to the same cohort of the bachelor program, they formed a relatively homogenous group. 
Students who discontinued their studies before earning the degree were not considered. 

Procedure 

As in our first study, we retrieved all necessary data for our second study from the Moodle database using 
SQL statements. The retrieval of the necessary data requires various steps. The tables in which the required 
information is stored must be identified. Since the documentation of the Moodle database is very limited, this can be 
very time consuming. Additionally, the SQL statement has to be defined. The following example demonstrates how 
we retrieved information about the students’ activities distributed by day of the week. 

1 . Three different tables (log table, user table, and course table) with the necessary information were 
identified and linked. 

2. The data needs to be limited to the students and courses described above. 

3. The results need to be grouped by day of the week. However, since all dates are stored in the UNIX 
time format (seconds elapsed since January 1, 1970), these data have to be converted to a regular date 
format. 

4. The SQL statement has to be defined (see Figure 2). 

5. The elements of the query have to be counted. 

Results and Discussion 

Place of Access to the LMS 

In order to identify whether students accessed the LMS Moodle from inside the university or from outside, 
we needed to find a sound indicator. An indicator for the place of access to the LMS is the internet protocol (IP) 
address. The IP address is assigned to devices (e.g. computer, laptop) connected to a computer network (internet). 
Accordingly, the IP address helped us to identify whether the student logged into the LMS Moodle from the 
university intranet with a specified IP addresses or from outside of the university. Figure 5 shows that access to the 
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LMS Moodle differed significantly between outside the university (M = 1850.78, SD = 1449.03) and inside the 
university (M= 282.33, SD = 205.66), f(17) = 5.148,p < .001, t/= .79. 




B Outside University 
B inside University 



Figure 5: Students’ place of access to Moodle 



Students ’ Activities 

In order to identify the distribution of access to Moodle within a week, we had to transform the UNIX time 
format to a standard date and time format. The grouped access times during the week are shown in Figure 6. The 
one-way ANOVA revealed a significant effect for access to Moodle between days of the week, F{6, 1 19) = 2.65, p < 
.05,/= .37. The average hits for the different days of the week were: Monday (M = 424.56, SD = 31 1.64), followed 
by Tuesday (M = 399.17, SD = 334.94), Wednesday (M = 380.50, SD = 364.42), Thursday (M = 294.94, SD = 

220.91) , Sunday (M = 250.67, SD = 189.88), Friday (M = 221.22, SD = 144.28), and Saturday (M = 162.00, SD = 

146.91) . 




Figure 6: Students’ access to Moodle by day of week 



Additionally, we were interested how the students’ access changed throughout the three years of their 
course of study. Table 2 shows the average scores and standard deviations of the students’ access to the LMS 
Moodle per semester. The one-way ANOVA revealed a highly significant effect between access to the LMS Moodle 
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per semester, F(5, 122) = 14.89, p < .001,/= .78. The students accessed the LMS Moodle least often in the first 
semester of the bachelor program, and they accessed it most often in the sixth and last semesters. 



Table 2: 


Average scores and standard deviations 
of access to the LMS Moodle per 
semester (A=18) 




Erequency 






Mean 


SD 


L' semester 


88.71 


89.94 


2"“* semester 


445.25 


317.98 


3^“* semester 


277.21 


140.49 


4* semester 


346.86 


168.57 


5* semester 


269.43 


104.21 


6* semester 


663.64 


455.35 



Discussion 

The selected findings of our second study revealed important information about the students’ use of and 
activities on the LMS Moodle. Over 86% of all hits to the LMS Moodle during the six semesters of the bachelor 
program occurred from outside the university. In our analysis of access to the LMS on different days of the week, 
we identified a decline in access towards the end of the week. Saturday was the day with the least student activity. 
Interestingly, the frequency of access to the LMS Moodle changed during the six semester course of study 
significantly. The least access was found for the first semester. Between the second and fifth semester the frequency 
of access increased greatly. For the last semester, we found the most frequent access to the LMS Moodle. To obtain 
even more detailed information about students’ activity in the LMS, we are conducting a further analysis to identify 
access times during the day and activities separated into different LMS modules. 

Study 3 

Studies 1 and 2 explored access to the LMS Moodle and usage of the available features. The very frequent 
access to the LMS Moodle (see study 2) indicates that the students are highly motivated to learn within the given 
learning environment. However, the application of available technological features is far from its limits (see study 
1). Therefore, our third study tried to identify a practicable taxonomy for designing effective online instruction 
within the LMS Moodle. 

Method 

Subjects 



Eight interviews with experts (3 female and 5 male) were conducted. Their mean age was 41.9 years (SD = 
10.2). All experts were experienced in using learning management systems for their courses. They had used Moodle 
for a mean of 7.6 semesters (SD = 2.7) and 19.4 courses (SD = 21.6). 

Procedure 

The structured online expert interviews consisted of four sections with closed-ended questions: (1) 
demographic data, (2) teaching experience, (3) user attitudes, and (4) instructional functions of Moodle features. In 
the section user attitudes, we asked the experts how often they used the Moodle features in their online courses. The 
following eleven Moodle features were included: assignment, chat, choice, forum, glossary, lesson, quiz, resources 
(files), survey, wiki, and workshop. Exclusive technological features (e.g. database, SCORM, etc.) were not 
considered in our analysis. The fourth section consisted of 77 pairwise comparisons in which we paired the above 
mentioned eleven Moodle features and seven possible instructional functions (information, communication. 
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cooperation, assessment, self-reflexion, feedback, evaluation) (see Hilgenstock & Jirmann, 2005). The interviewees 
were asked to rate the extent to which each pair, e.g. assignment - information, was applicable to the instructional 
function. Additionally, we asked the experts how confident they were concerning each pairwise comparison. 

The collected data was transcribed and categorized for further statistical analysis. In order to build a 
practicable taxonomy, we are looking for Moodle features that are highly applicable to instructional functions. 
Additionally, these results have to be supported by a high confidence level on the part of the experts. 

Results and Discussion 

Usage of Moodle Features 

The analysis of the experts’ usage of Moodle features corresponds in part to the results of our first study. 
All experts stated that they use the feature //Ze {M = 5.00, SD = 0.0). The feature /or«m is also used in their online 
courses very often (M = 4.75, SD = .46). The features assignment (M = 3.75, SD = 1.16), glossary (M = 3.00, SD = 
1.51), quiz (M = 3.00, SD = 1.60), choice (M = 2.88, SD = 1.64), and wiki (M = 2.88, SD = 1.64) are used 
moderately. The features workshop (M = 2.13, SD = 1.64), lesson (M = 2.38, SD = 1.89), chat (M = 2.13, SD = 
1.55), and survey (M = 1.88, SD = .99) are used very seldom (see Figure 7). 




Figure 7: Usage of Moodle features (1 = never, 5 = very often) 



Pairwise Comparisons 

The results of all 77 pairwise comparisons between the eleven Moodle features and the seven instructional 
functions are presented in Table 3. The experts’ confidence (1 = low confidence, 5 = high confidence) with regard to 
their pairwise ratings was relatively high (M = 3.66, SD = .84). Hence, we conclude that the experts’ knowledge of 
the Moodle features and the instructional function was adequate for our study. 
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Table 3: 



Average scores (standard deviations in parenthesis) of pairwise 
comparison between Moodle features and instructional functions rated by 
experts between 1 (minimal) and 1 1 (maximum) applicability (N = 8) 





information 


communicatioi 


cooperation 


assessment 


self-reflexion 


feedback 


evaluation 


assignment 


7.50 


4.38 


6.63 


10.13 


9.88 


9.38 


8.75 


(3.67) 


(3.25) 


(2.77) 


(.99) 


(.99) 


(1.06) 


(3.41) 


chat 


6.50 


10.63 


7.75 


3.88 


6.75 


7.88 


4.67 


(3.12) 


(.52) 


(4.37) 


(3.76) 


(4.10) 


(2.48) 


(2.42) 


choice 


8.50 


7.88 


6.50 


6.67 


5.88 


8.38 


7.75 


(3.30) 


(2.17) 


(3.07) 


(3.88) 


(2.30) 


(1.77) 


(2.92) 


files 


11.00 


4.63 


4.63 


2.75 


4.75 


2.50 


2.13 


(.00) 


(3.70) 


(4.00) 


(2.44) 


(3.24) 


(2.20) 


(1.81) 


forum 


8.63 


10.86 


10.13 


5.00 


9.50 


10.13 


5.38 


(2.56) 


(.38) 


(.84) 


(3.70) 


(1.38) 


(.99) 


(3.89) 


glossary 


10.63 


6.00 


8.88 


4.63 


6.25 


4.13 


5.25 


(.52) 


(4.07) 


(3.36) 


(2.67) 


(3.28) 


(4.02) 


(3.26) 


lesson 


9.38 


2.57 


4.00 


6.43 


4.63 


6.25 


6.25 


(1.69) 


(2.07) 


(2.27) 


(2.23) 


(2.07) 


(2.49) 


(3.33) 


quiz 


3.13 


3.38 


3.13 


10.50 


8.57 


10.00 


9.88 


(3.40) 


(3.89) 


(4.02) 


(.54) 


(2.23) 


(1.31) 


(1.73) 


survey 


5.38 


6.00 


3.38 


3.88 


4.88 


6.63 


6.50 


(3.46) 


(2.89) 


(2.83) 


(2.59) 


(2.90) 


(3.89) 


(2.51) 


wiki 


9.75 


8.50 


10.25 


5.25 


9.63 


9.57 


5.13 


(1.28) 


(3.07) 


(.89) 


(3.37) 


(1.19) 


(1.51) 


(3.36) 


workshop 


7.00 


8.00 


9.25 


7.00 


9.17 


6.83 


7.50 


(2.83) 


(2.14) 


(1.49) 


(2.97 


(1.94) 


(3.76) 


(2.07) 



Discussion 



The findings of the third study helped us to identify a practicable taxonomy for designing effective online 
instruction within the LMS Moodle. Using the results of the pairwise comparisons and the confidence ratings, we 
identified specific Moodle features that are applicable for designated instructional functions (see Table 4). 
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Table 4: Applicability of Moodle features for designated instructional functions 





information 


communication 


cooperation 


assessment 


self-reflexion 


feedback 


evaluation 


assignment 








✓ 




✓ 




chat 
















choice 


✓ 


V 








V 




files 


✓ 














forum 




✓ 


✓ 






V 




glossary 


✓ 














lesson 


✓ 














quiz 








✓ 




✓ 




survey 




✓ 






✓ 




wiki 


✓ 




•/ 




•/ 






workshop 




✓ 






•/ 







General Discussion 

Both quantitative analysis of the SQL database and qualitative analysis of interview protocols were used in 
the tradition of triangulation to strengthen the findings of our study on LMS. Although learning management 
systems have great technological and instructional potential, most of the available features are not used by course 
creators and instructors. Being aware of this instructional regression, we demand a taxonomy of common LMS 
modules. This taxonomy should include a definition of the available LMS features, their key purpose, benefits for 
learning and teaching, and a guideline for course instructors. 

Following the taxonomy of common CMS features related to resource type and value by Koszalka & 
Ganesan (2004) and on the basis of our empirical findings, we developed a practical taxonomy of the above 
discussed Moodle features. Table 5 shows the taxonomy, which contains (1) the name of the Moodle feature, (2) a 
short description of the Moodle feature, (3) a classification of the Moodle feature with regard to the three design 
elements information, instruction, and learning (see Grabowski & Curtis, 1991), (4) the Moodle feature’s 
association with instructional functions (see Hilgenstock & Jirmann, 2005), and (5) exemplary recommendations for 
instructional use. 

However, further research studies are needed to validate this taxonomy. In addition, using all of the 
available features of an LMS will not necessarily improve the learning process, an aspect which will be investigated 
in future projects. 
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Table 5: Practicable taxonomy of Moodle features 
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Abstract 

The purpose of the present study is to develop an integrated motivation scale to measure the level of motivation for 
employees in e-learning. This paper reports on the validation process of the scale by administrating the scale to 4392 
employees in e-learning. The new motivation scale is composed of four factors: 1) self-regulated learning; 2) 
perceived choice; 3) online task value; and 4) self-efficacy. A follow-up study is in progress to identify the 
relationship between motivation status and learning outcomes. 

Introduction 

There are many studies that have developed a motivation scale in face-to-face instruction (Pintrich & DeGroot, 

1990; Zimmerman & Martinez-Pons, 1986; Deci & Ryan, n.d.) and a few studies have been conducted on e-learning 
setting (Lee, 2003). However, it is difficult to find a motivation scale which considers the corporate environment. 
This study identified motivation with learners’ energy, drive to learn and the behaviors which follow from this 
energy and is focused on measuring motivation factors which complete an e-learning course continuously, 
successfully and efficiently. 

The purpose of the study was to develop a motivation scale to measure the level of motivation for employees in 
corporate e-learning. This study reports the reliability and validity of the new scale— l)self-regulated learning, 2) 
perceived choice, 3) online task value, and 4) self-efficacy — in a corporate environment. 

Theoretical Framework 

From the literature review, background variables that could influence the effectiveness of e-learning are as follows: 
Self-regulated learning 

Recently, self-regulation is considered as one of the importance characteristics of online learners (Hodges, 2005). 
Learners who are good at self-regulation use various strategies to learn effectively (Zimmerman & Martinez-Pons, 
1990). Thus, for self-regulated learners, many types of self-regulated learning strategies are required. 

Perceived choice 

Individual intrinsic motivation is determined by the concept of self-determination related to perception of the 
guarantee with the autonomy in the monitoring or regulating of activities (Ryan & Deci, 2000). Following the self- 
determination theory (SDT), higher self-determination is related to higher intrinsic motivation (Vallernad et ak, 
1992). Therefore, the choice made by a learner’s determination, that is perceived choice, was important in a learning 
circumstance. 

Online Task value 

Online Task value is composed of interest, utility, and importance (Wigfield & Eccles, 1992). Researchers have 
found that adolescents’ subjective task values predicted both their actual and anticipated task choice. According to 
Bong (2001), task-value predicted both students’ performance and enrollment intentions. 
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Self-ejficacy 

Self-efficacy is defined as “belief in one’s capabilities to organize and execute the courses of action required to 
produce given attainments” (Bandura, 1997). According to Joo, Bong, and Choi (2000), academic self-efficacy 
affects various aspects of student motivation and achievement. 

Methods 

Participants 

The learners who were enrolled in e-learning courses offered at an employee education center in the fall of 2007 
participated in the study. This study was administered to 4547 learners enrolled in the course, and 4392 of the 
learners participated in the study. This course lasted for five week. For items analysis, EFA was performed with 
1781 participants; CFA was conducted with the remaining 2611 participants. 

Procedure 

Study 1 

Based on the theoretical review of learners’ motivation, we focused on four components of motivation. The 
theoretical framework of each component is summarized in Table 1. 

Table 1 Theoretical framework of the four components of motivational status. 



Components 


Research support 


Self-regulated learning 


Zimmerman & Martinez-Pons(1886), Lee(2003) 


Perceived Choice 


Deci & Ryan( n.d.) 


Online Task Value 


Bong(2001) 


Self-efficacy 


Pintrich & De Groot,(1990) 



We developed 26 items with a 7-point Likert scale. The items were analyzed by an expert for content validity and 
modified based on recommendations to fit e-learning circumstances. An online survey was conducted and 4392 
responses were collected. With 1781 participants, an EFA was performed to verify the emergence of 
four dimensions of motivation. The principle axis factoring method was used to extract factors. Kaiser’s rule and 
scree test with visual inspection was also used to determine the number of factors to be extracted. To rotate factors, 
the direct oblimin rotation method was used. Factor loading of the items was higher than .30 in standard. 

Study 2 

To confirm construct validity, a CFA was conducted with the remaining 2611 participants. Based on the result of 
EFA in Study 1, the model was developed by using AMOS 6.0. To evaluate model fitness, TLI (Tucker-Lewis 
index), CEI (Comparative fit index), and RMSEA (Root mean square error of approximation) not 
affected by the number of sample (Hong, 2000) were used as a fit index. 

Results 
Study 1 

The results of the EEA with 26 items yielded four factors: self-regulated learning, perceived choice, online task 
value, and self-efficacy. The four factors showed the reliability with Cronbach’s coefficient alpha .922, .708, .891, 
.889, respectively. The three items that took double loading were removed; consequently 23 items were retained 
(Table 2, 3 ). 



148 




Table 2 Factor loading for items 



Factor 




1 


2 


3 


4 


SRL5 


.777 








SRL4 


.769 








SRLl 


.736 








SRL6 


.729 








SRL2 


.727 








SRLIO 


.723 








SRL8 


.703 








SRL7 


.686 








SRLll 


.686 








SRL12 


.656 








SRL13 


.647 








SRL9 


.558 








PC3 




.866 






PCI 




.652 






PC22 




.564 






OVAL4 






-.866 




OVAL2 






-.856 




OVAL3 






-.848 




ASE5 








-.847 


ASE2 








-.823 


ASE3 








-.774 


ASEl 








-.759 


ASE4 








-.710 
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Table 3 Retained items of the new scale 



Constructs 


Items 


Cronbach’ s 
a 




1 . 1 check over my work to make sure I did it right. 

2. 1 took notes of the class discussion. / 1 kept a list of the words I got wrong. 

4. 1 make an outline before I write my paper. 

5. Eirst, I start studying two weeks before exams, and I pace myself. 

6. In preparing for a math test, I keep writing the formula down until I remember it. 

7. Before beginning to write the paper, I go to library to get as much information as 




Self- 


possible concerning the topic. 




regulated 


8. 1 isolate myself from anything that distracts me. / 


.922 


Learning 


I turn off the radio so I can concentrate on what I am doing. 

9. If I have problems with math assignments, I ask a friend, a teacher, and other 
adults to help. 

10. When preparing for a test, I review my notes. 

1 1. 1 log on my e-learning courses regularly. 

12. 1 manage my e-learning schedule not to interfere with other work. 

13. 1 submit my assignment or learning activity on time. 




Perceived 

Choice 


1 . 1 felt like it was not my own choice to do this task. 

2. 1 did this activity because I wanted to. 

3. 1 believe I had some choice about doing this activity. 


.708 


Online 
Task Value 


1 . 1 find this online course interesting. 

2. 1 think what I learn in this online course is important. 

3. 1 think this online course is a useful course. 


.891 



1. 1 am sure I can do an excellent job on the problems and tasks assigned in this 



class. 

2. 1 know that I will be able to learn the material for this class. 

Self-efficacy .889 

3. 1 expect to do very well in this class. 

4. 1 think I will receive a good grade in this class. 

5. 1 am certain I can understand the ideas taught in this course. 



Study 2 

In order to confirm three constructs obtained in Study 1, a CFA was conducted. TLI, CFI, and RMSEA were used 
to evaluate model fitness in this study. As a result, CFI=.930 means a good fitness, TLI=.921 also means a good 
fitness, and RMSEA=.064 means a reasonable fitness (Hong, 2000). Table 4 shows the suitable numerical indexes 
that are accepted from CEA about the model. 



Table 4 CEA result 



Model 




df 


P 


CPI 


TLI 


RMSEA 




2671.056 


224 


*** 


.930 


.921 


.064 
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Conclusion 

This measure of learners’ motivation in e-learning is theoretically grounded and went through a validation process. 

As expected, employees’ motivations in an e-learning environment is composed of four dimensions: 1) self- 

regulated learning’ 2) perceived choice; 3) online task value’ and 4) self-efficacy. The validity for each construct 

was confirmed by CFA in Study 2. 

This study will serve as a first step in measuring educational effectiveness in employee online education. Currently, 

data collection is in progress to identify the relationship between motivation status and learning outcomes. 
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Abstract 

Research and program evolution studies examining the effectiveness of training simulations in the transfer 
of skills under high risk confrontational (HRC) settings is scarce and is the driving factor behind the present study. 
High risk confrontational settings are defined as any job or task that requires placing oneself in a life-threatening 
confrontation with element(s) or another person(s). Those working in HRC settings experience high stress levels in 
their jobs and are known to have decreased performance and potentially decreased survival. To effectively and 
safely accomplish their job, individuals must be trained to handle such conflicts. Although simulations have been 
praised for their training effectiveness, some studies report little or no benefit of using simulation to facilitate 
learning. Lack of empirical support is particularly apparent in simulations developed to supplement training of 
individuals who work in HRC settings. Any correction in the simulation training arena would lead to a higher 
potential of success in both performance and survival. 

Two primary reasons account for this scarcity of empirical findings: (a) the cost(s) associated with 
adequately replicating the dynamic situations found in HRC settings, and (b) the fact that simulations are usually too 
focused, with a specific objective supporting a finite set of conceivable options with a finite set of reactions. The 
present study focused on examining novice players and the transfer of a skill (target acquisition) measured by an 
increase in performance in a real environment (RE) gained from exposure to a virtual environment (VE). In order to 
measure transfer, two interventions were applied; interventions consisted of a desktop computer-based paintball 
game (the simulation treatment) and a paper-based manual on shooting fundamentals (the non-simulation 
comparison application). 

Upon receiving the interventions, participants met two-days later at an outdoor facility for two live practice 
sessions (game 1 and 2) of paintball. Participants were asked to mentally track the number of direct hits made on 
opposing participants, reporting the number upon exiting the field. Time on the field divided by the number of 
opponents hit was used to calculate target acquisition per participant. A one-way repeated-measures ANOVA with 
one between-subjects factor was conducted to investigate the question as to whether or not there was a difference 
among intervention received. The within-subjects variables were observations made from game 1 and game 2. The 
between-subjects factors were the simulation treatment and the non-simulation comparison application. 

Overall, findings show a statistically significant difference between novice players in the type of 
intervention received and target acquisition score. Reasoning behind the significance may be attributed to cognitive 
overload; however, the two interventions used different cognitive applications. The simulation treatment is 
classified as a mechanical cognitive process, whereas the non-simulation comparison application is classified as a 
verbally based cognitive process. Those receiving the simulation treatment may have performed better from game 1 
to game 2 because of the mechanical application being more experiential and, therefore, easier to assimilate than the 
verbal application of text alone. Since there is a link between the interventions (learning) and increased performance 
(target acquisition), transfer of learning can be argued. Eor those participants who did not score a hit (in either game 
or those who scored hits in game 1 but not game 2), it can be argued that negative transfer was experienced in 
response to aiming and firing. On the other hand, for those participants who received the simulation treatment and 
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did score hits (in either game); it can be argued that complex transfer was experienced. Furthermore, for those 
participants who received the non-simulation comparison application and did score a hit in either game, far transfer 
can be argued. 

The significant difference in target acquisition scores and the type of intervention received hold 
implications for the type of training applied to those learning skills for HRC settings. Communities of practice 
include practitioners, instructional technologists, and researchers. For the practitioner, the advantages of using a 
desktop simulation, rather than text-based non-simulated material, should be considered for tasks that are physically 
demanding, dynamic in nature, and involve complexity and risk. As an instructional technologist, selection of an 
appropriate use of a desktop computer simulation should be considered as a type of medium to use for HRC training 
applications to decreasing the learning gap and increasing safety by advancing learners from basic knowledge to 
practical application. If an intervention presents a greater capability to aid in this transfer, these methods need to be 
considered and applied appropriately. Finally, the research community should continue to explore and quantify 
desktop simulations as a training medium and explore the variables that would provide the greatest effectiveness of 
transfer in learners. 
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Introduction 

High risk confrontational (HRC) settings are defined as any job or task that requires placing oneself in a 
life-threatening confrontation with element(s) or another person(s). Those working in HRC settings experience high 
stress levels in their jobs and are known to have decreased performance and survival (Atkins & Norris, 2004). 
Individuals being trained to handle such conflicts should be trained effectively in successfully and safely 
accomplishing the objectives related to their job. Although simulations have been praised for their training 
effectiveness (Paul, Eleig, & Jannin, 2005), some studies report little to no benefits of using simulation to facilitate 
learning. Eor example, the Department of Defense (DoD) study. Department of Defense Plan for Integrating 
National Guard and Reserve Component Support for Response to Attacks Using Weapons of Mass Destruction 
(1998), reported that a gap exists between battlefield skills training for first responders and the unique response 
skills training required for civilian first responders. 

Lack of empirical support is particularly apparent in simulations developed to supplement training of 
individuals who work in HRC settings. This scarcity may be due to the costly nature of running HRC events and the 
difficulty in creating all possible dynamic situations (Cloud & Rainer, 1998). In addition, research conducted under 
stressful situations is broadly categorized to examine physiological responses, team dynamics, and decision making 
and not necessarily focused on skill acquisition. Delivering training through simulation is a common approach for 
those organizations, communities, and industries responsible for training difficult and complex tasks. 

The present study addresses this scarcity of research on skill acquisition by examining whether or not 
target acquisition skills can be transferred to a real environment (RE) under HRC settings after exposure to those 
skills within a virtual environment (VE). Specifically, the present study investigates if there is increased 
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performance in target acquisition for novices who were exposed to a simulation treatment (represented in a desktop 
computer-based paintball game) prior to performing in a RE (represented in a live paintball game) as opposed to 
novices who were exposed to a non-simulated comparison application (represented as a manual on shooting 
fundamentals). Three hypotheses were tested: (a) HoP There is no significant difference in target acquisition scores 
for individuals who received the simulation treatment and those who received the non-simulation comparison 
application while performing in an HRC setting, (b) H 02 : There is no significant change in target acquisition scores 
from game 1 to game 2 based on practice while performing in an HRC setting, (c) H 03 : There is no interaction 
between type of intervention received and practice in an HRC setting. 

Ability Determinants of Skill Acquisition 

Given that the task of target acquisition is complex and dynamic, where initiating cues and consequences 
of actions vary extensively, and that target acquisition can be replicated with sufficient fidelity to overlap with a RE, 
Ackerman’s (1988) Theory of Ability Determinants of Skill Acquisition was adopted as the theoretical foundation for 
the present study. The theory requires that tasks be complex in nature and have inconsistent information processing 
while containing substantial overlap to a RE. Ackerman (1992) explains that there are three phases in the transfer of 
learning for skill acquisition: (a) cognitive, (b) associative, and (c) autonomous. Collectively these three levels 
define a cognitive process that distinguishes a novice from an expert. The phases of the process build upon one 
another to the point that skill-based behavior eventually becomes automated in response to complex environmental 
stimuli, as can be found in HRC settings. 

Based on the definitions of Ackerman’s (1992) phased approach, the cognitive phase is focused on 
formulating concepts and developing procedural skill, such as attention to semantics for verbal information related 
to the text-based description or spatial orientation for maneuvering within a simulation. During the associative 
phase, basic skill and knowledge becomes engrained. There is less deliberate cognitive focus and more of an 
emphasis on increasing speed and accuracy through practice. With continued practice, the novice moves toward 
mastery or the autonomous phase exemplified by expert behavior. (This concept of practice is important, as the 
hypotheses presented in the present study focus on such.) In this phase, actions are automatic and the focus is on 
refining psychomotor responses. 

Ackerman’s (1998 and 1992) cognitive model was modified to fit the needs of the present study — shown 
in Eigure 1 . The skill of target acquisition by novice players begins at the cognitive phase where the task of finding, 
aiming, and shooting are introduced with written information — shown in Eigure 1 as the Verbal (Non-Sim) 
independent variable (IV) — or in a computers-based format — shown in Eigure 1 as the Mechanical (Simulation) IV. 
Because participants are novice players, there will be a high cognitive demand once in the RE. With practice, a 
transition to the associative phase of increased speed and performance begins during actual target acquisition, 
shown in Eigure 1 as the Performance (Target Acquisition) dependent variable (DV). It is anticipated that some 
participants will not experience the associative phase due to limited exposure to the RE. Other rationale for 
remaining in the cognitive phase lies with tasks that have inconsistencies requiring greater attentional resources that 
do not become easier over time. According to Ackerman (1992), these tasks do not transition to the associative 
phase because they remain attention dependent. 
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Figure 1. Modified Representation of Ackerman’s (1998 and 1992) Cognitive Model 

Empirical Support 

Four areas of research provide the empirical foundation for the present study. These areas include: (a) the 
RE of live paintball play, (b) the VE for training, (c) the key concepts and elements of learning transfer, and (d) 
target acquisition research and measurement. These four areas are relevant to the present study in the following 
ways. Eirst, the RE is represented by a live paintball setting. Although research on the use of live paintball as a 
method for teaching is limited, there is evidence to show that live paintball can play a major role in creating a sense 
of realism (Correll, Park, Judd, Wittenbrink, Sadler, & Keesee 2007; Gordon, 2005; Jacobellis, 2007; Smith, 2003). 

Second, the VE is represented in a simulated desktop computer-based game. Studies that focus on a 
comparison of a VE to a RE training method have been limited. But studies that compare a desktop computer to 
other simulated environments (e.g., a head mounted display (HMD) or fully-immersive CAVE environment with 10 
foot walls and an HMD) have shown that the use of desktop computers as virtual environments lead to effective 
transfer (Loftin, Scerbo, McKenzie, & Catanzaro, 2004; Moreno & Mayer, 2004). 

Third, transfer of learning is represented in three dimensions to include: (a) positive and negative transfer, 
(b) simple to complex transfer, and (c) the occurrence of near and far transfer of why transfer may or may not occur. 
Positive transfer facilitates learning or performance and occurs when stimuli and responses are similar; however, 
negative transfer means that a learned response actually hinders appropriate performance (Leberman, McDonald, & 
Dole, 2006; McKeachie, 2001; Royer, 1986). Simple transfer, occurs when previous knowledge can be used in a 
new situation with little to no effort. Complex transfer is defined as using the previously acquired knowledge in a 
new situation while seeking extended applications in which that knowledge can be used (Leberman et. al, 2006). 
Near transfer is posited to take place when previous knowledge is being applied to situations that are similar to what 
is being newly experienced and takes minimal cognitive effort (Leberman et al., 2006; McKeachie, 2001; Royer, 
1986). Far transfer is essentially the process of applying existing knowledge to a novel learning situation which 
takes a high cognitive effort (Leberman et al., 2006; McKeachie, 2001). 

Einally, research on the measurement of target acquisition is limited. One of the few articles on the use of 
ammunition and delivery of fire, presented by Eresenko (2002), defines target acquisition as the relationship 
between ammunition consumption and the degree of damage. For the purpose of the present study, only the degree 
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of damage is measured by the number of opponents hit and the time in live play prior to being hit. A standard 
mathematical formula, endorsed by an NRA Certified Pistol Instructor (Dwyer, 2008), was used to provide a ranked 
scoring method. 



Method 

Participants and Design 

A posttest-only randomized comparison group research design was adopted for the present study. The 
voluntary participants were 24 individuals (19 males and 5 females) over the age of 18 working in a major research 
center and university in the southeastern region of the United States. A stratified sampling procedure was used to 
represent the distinct categories of novice paintball players with no experience in playing live paintball. All 
participants were randomly assigned to two groups. Fifty percent of the participants were exposed to the simulation 
treatment, while the other fifty percent were exposed to the non-simulation comparison application. One of the 
participants was not able to continue the second-half of the study, leaving the final participant count at 23. All 
participants were treated in accordance with the American Psychological Association’s Ethics in Research with 
Human Participants (1992). 



Materials and Apparatus 

Two instruments were used in the present study, a classification matrix and reaction questionnaire. The 
classification matrix was provided as initial requirements to participate in the present study, while the reaction 
questionnaire was administered after intervention was received and again after game 2 of the live paintball sessions. 
As for the interventions, a simulated desktop computer-based game was used to determine if prior exposure to the 
VE had a positive effect on target acquisition, while a paper-based manual on shooting fundamentals was used for 
comparison and interaction effect. 

Classification Matrix 

Eor stratification and to eliminate differences between groups, participants were asked to complete a 
classification matrix prior to the start of the present study. The matrix focused on a self-rating by participants 
regarding their own computer skills, physical ability, and shooting ability. Participants were instructed to “Select the 
best answer that applies by placing an ‘X’ in the appropriate box” followed by a list of three questions: “Do you 
consider yourself to have high or low computer skills?” “Do you consider yourself to have a high or low physical 
ability?” and “Do you consider yourself to have a high or low shooting ability?” Participants choose between “high 
computer skills” and “low computer skills”, “high physical abilities” and “low physical abilities”, and “high 
shooting abilities” and “low shooting abilities”. The classification matrix also collected age demographics by asking 
participants to “Select your appropriate age range”. Participants choose either “18 - 30” or “35 - older”. 

Reaction Questionnaires 

Reaction questionnaires provided the opportunity for participants to rate their own target acquisition 
ability, to indicate the treatment they received, their opinion about the benefit of the type of treatment received 
compared to the live practice sessions in the RE, and to rate the mental challenge of the treatment received as well 
as the mental challenge during live practice sessions. Two reaction questionnaires were administered during the 
present study. A training methods reaction questionnaire was collected after the intervention, while a live play 
reaction questionnaire was administered after the final practice session (game 2) was played in the RE. 

Eor the training methods reaction questionnaire, participants were instructed to “Please select the response 
for each item that best reflects your opinion.” Participants were asked “Have you ever practiced shooting at a target 
in the past?” and could reply with “no” or “yes”; “If yes, how accurate do you feel you were at hitting your target?” 
and could reply with “very accurate”, “accurate”, “neither accurate nor inaccurate”, “inaccurate”, or “very 
inaccurate”; “Which training method did you receive?” and could reply with “pc game” or “text-based”; “Do you 
feel the training method you received will be of help to you during the live play sessions?” and could reply with 
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“no” or “yes” and could specify why; and “Was the training method you received mentally challenging?” and could 
reply with “no” or “yes” and could specify why. 

For the live play reaction questionnaire, participants were instructed to “Please select the response for each 
item that best reflects your opinion.” Participants were asked “Have you ever practiced shooting at a target in the 
past?” and could reply with “no” or “yes”; “How accurate do you feel you were at hitting a target on the field during 
live play?” and could reply with “very accurate”, “accurate”, “neither accurate nor inaccurate”, “inaccurate”, or 
‘very inaccurate”; “Which training method did you receive?” and could reply with “pc game” or “text-based”; “Do 
you feel the training method you received was of help to you during the live play sessions?” and could reply with 
“no” or “yes” and could specify why; and “Were the live paintball sessions mentally challenging to play?” and 
could reply with “no” or “yes” and could specify why. 

Simulation Treatment 

The VE encompassed a simulated desktop computer-based game used to emulate a group paintball 
skirmish and served as the simulation treatment for Group A. The game was a commercially available paintball- 
themed PC -based computer game running on a Dell Dimension 280, Pentium 4 at 3.20Ghz, 1GB RAM, 80GB HD 
system running Windows XP SP2 with a 17 inch USB Dell Monitor running at 1024x768 resolution. Participants 
used a standard 101 Dell keyboard and USB optical three-button Dell mouse for game play (default controls were 
used). Headphones were provided. 

Although the desktop game was pre-configured to mimic the type of field used during the two live practice 
sessions, there were some unique characteristics of the simulation. These were special computer generated aspects 
not possible with a RE. These included a flying segment before the start of each game, a radar view (map) where 
opponents could be located, and a back-to-life option following a hit. Although some features of the simulation 
were not realistic, the basic actions within the game of finding, aiming, and shooting closely resembled live game 
play. 

Non-Simulation Comparison Application 

A paper-based manual on shooting fundamentals served as the non-simulation comparison application. The 
manual consisted of eight-pages of text with graphics depicting the handling and firing of a pistol. The material was 
taken from Chapter 6, “Shooting Eundamentals,” from the Basics of Pistol Shooting by the National Rifle 
Association (1991). Shooting fundamentals are the baseline guidance for proper pistol application to acquire a 
target. 

Live Paintball Eacility, Gear, and Equipment 

The RE encompassed an outdoor arena for live game play located at a commercial paintball facility. 
Although multiple fields were available for use, only one field was used for the present study. The field (see Eigure 
2) was setup with numerous obstacles (made from corrugated metal) serving as cover for participants during the two 
live practice sessions (games 1 and 2). 
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Figure 2. Live paintball facility field depicting camera positioning 



Gear for each participant consisted of (a) helmets, (b) weapons, (c) and paintballs. Protective head gear 
included a JT XFire goggles elite mask system. Weapons consisted of semi-automatic Spider Extra and Tippman 98 
paintball guns commonly referred to as a marker. These semi-automatic guns are capable of releasing a paintball 
upon the pull of the trigger with a splatter zone of 50 to 75 yards effectively. Paintballs consisted of a gelatin casing 
covering a crayon wax filling with a water-based food coloring for detection. 

Several hours prior to live game play, video equipment was stationed in three locations for later viewing 
and verification of time players were hit. Cameras were strategically positioned on the playing field to avoid 
obstruction to participants (see Figure 2). Camera one was a Sony HDV Zl, camera two was a Sony HDV FXl, 
while camera three was a Sony Mini-DV. All video footage was later compiled by a video subject matter expert into 
three video scenes synched together for simultaneous viewing using the Final Cut Pro editing software. 

Procedure 



Recruitment 

Participant recruitment was conducted through email. To take part, participants had to sign a consent form 
and complete the classification matrix. Consent forms and classification matrices were returned by either email as a 
scanned attachments or facsimile. 

Simulation Game and Paper-based Manual 

Once all consent forms and classification matrices were received, participants were randomly assigned to 
one of two groups. Participants arrived at a local university for the first-half of the study which took place two days 
prior to live game play. Each was assigned an alphabetical letter in the order in which they arrived. This letter was 
later used to provide the participants with T-shirts for identification purposes during the live practice sessions. 

Group A, randomly assigned the simulation treatment, engaged in the 10-minute simulated desktop 
computer-based paintball game. Participants were allowed to sit in front of any of the 5 computer workstations 
running the desktop simulation. The simulation game was repeated after a 5-minute break period. Upon completion 
of the second game, participants were asked to fill out the treatment methods reaction questionnaire. At the same 
time. Group B, randomly assigned to the non-simulation comparison application, read the 10-minute non-simulated 
paper-based manual on shooting fundamentals. Participants were allowed to sit in front of any desk setup with the 
manual. Participants were asked to read the manual a second time after a 5-minute break period. Upon completion 
of the second reading, participants were asked to fill out the treatment methods reaction questionnaire. Prior to 
leaving, each participant was verbally informed not to participate in any paintball activity between receipt of the 
intervention and the live practice sessions. Each participant was also told they would receive an email notification 
that provided the time, date, and directions to the paintball facility. 
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Live Practice Sessions 



The live paintball engagement took place at the paintball facility two days after receipt of the intervention. 
Participants were provided assigned T-shirts to wear for identification purposes while on the field and for review of 
the video recordings. As participants were lined up to receive their paintball safety gear, they were randomly 
assigned to a team by calling out a “1” or a “2” consecutively while in line. Team 1 was distinguished with colored 
tape placed on their helmets, while team 2 wore plain helmets. All participants received basic instructions on how to 
play by the paintball facility referees to ensure game safety. Each participant received approximately 200 paintballs 
for use during each live practice session. Additional paintballs were supplied during break periods as needed. 

During each practice session, the participants performed a series of events in which they sought cover, 
scanned for opposing participants, and attempted to eliminate the opposing participant from the game by striking 
them with a paintball. Participants directly hit by an opposing team member were instructed to leave the field. 
Referees, supplied by the paintball facility, ensured these rules were followed during game play. Upon completion 
of game 2, participants were asked to fill out the live play reaction questionnaire. 

There was an hour break between practice sessions. This break period was essential in determining if the 
treatment or comparison application were carried into the RE. It consisted of the participants’ choice to engage in 
additional sessions with any player who was at the facility, take a break from playing, or drop out of the game. 
Information about what the participants engaged in during the break period was noted on the data collection sheets. 
If the participant engaged in additional live play, the number of sessions played for each participant was recorded. 
This information was later used for additional statistical analysis. 

Scoring 

Each participant was asked to keep mental track of the number of direct hits made on opposing participants 
during each practice session. To account for a closer approximation of actual time hit, the referees ensured 
participants, who had been hit, did in fact leave the field immediately; leaving fewer participants on the field over 
time. Upon exit, each participant verbally provided the number of opponents they perceived they hit to raters 
stationed at the field exit. These raters were used to ensure reliability of the scoring during the two practice sessions. 
These numbers were hand-recorded on data collection sheets next to the participant’s assigned alphabetical letter. 

Video records of the two practice sessions were used for later analysis. This analysis was predominately 
used to confirm hits self-reported by the participants and the time when each participant was hit and subsequently 
exited the field. To ensure reliability of the scoring, independent raters reviewed the videos. Because the actual 
paintballs were difficult to view in the videos due to outdoor lighting conditions, the method used to determine time 
hit by the raters was based on the participant raising their hand and/or verbally calling out, “I’m hit!’’ on video. 

(This behavior was taught as part of the training given by the paintball facility referees.) Consequently, the 
recordings were collaboratively reviewed multiple times. 

The numbers taken from the verbally provided number of opponents hit and video recordings for time on 
field were then used in the calculation of target acquisition (participant’s performance) scores. Target acquisition 
was calculated as the number of actual hits on opposing participants based upon the amount of time the participant 
lasted within each live practice session. The formula is as follows: 

Target Acquisition = Time on Eield (in seconds) / Number of Opponents Hit 

The lower the participant’s score the higher the participant ranks in target acquisition. Eor example, if a 
participant hits two opponents in 46 seconds prior to being eliminated, he or she will have a ranking of 23. If 
another participant hits 4 opponents but remains in play for 154 seconds prior to being eliminated, he or she will 
have a ranking of 39 (consideration of rounding). This formula was designed to help rank participants for analysis 
purposes only and does not suggest that the lower ranking participants actually performed better or were more 
efficient than higher ranking participants. To summarize, each participant was paired with his or her score of targets 
acquired (the verbal report by the participant of the number of targets acquired when leaving the field) with the time 
recorded on video footage from the two live practice sessions. 
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Results 



Participant Demographics 

The majority of participants were male (n=19) comprising all of Group A and over half (58.3%) of Group 
B. This left all females (n=5) in Group B. The majority {bb.1%) of participants in Group A were between 18 and 30 
years of age, whereas the majority (75%) of participants in Group B were 31 years of age or older. 

With regard to the classification matrix, groups A and B included a fairly even distribution in computer 
skill, but not in physical and shooting ability. Those in Group A self-rated their computer skills high at 83.3%, 
whereas Group B self-rated at 75%; all in Group A self-rated their physical ability high, whereas only 66.7% did so 
in Group B; and finally, 75% of those in Group A self-rated their shooting ability as high, compared to 41.7% in 
Group B. 

A Pearson Correlation was performed to determine if a relationship between demographics and target 
acquisition scores existed (which represented the ability of an individual to perform better during the live practice 
sessions). Results suggest that there is no relationship between gender, age, physical or shooting ability and target 
acquisition scores. Mean scores for game 2 were higher (M = 73.86, SD = 74.32) than mean scores for game 1 (M = 
43.29, SD = 49.23). However, there was a statistically significant negative correlation (-0.449, p = 0.036) between 
gender and scores taken from game 2, suggesting that as male scores increased, female scores decreased. 

Of the total participants, four males and one female were excluded from the analysis due to missing data. 
The excluded participants were descriptively similar to those included in the final dataset with an 80% male and 
20% female composite. However, with regard to age, all excluded from the dataset were in the younger age bracket. 

Primary Results 

A one-way repeated-measures ANOVA with one between-subjects factor was conducted to investigate 
whether a virtual environment can effectively be used to train individuals to perform better in HRC settings. An 
alpha of .05 was used. The within-subjects variables were the two live practice sessions. The between-subjects 
factors — the intervention levels — were the simulation treatment and the non-simulation comparison application. 
Results of game 1 and game 2 represent the set of target acquisition scores based on the live practice sessions in the 
RE. 

Hypothesis I 

There is no significant difference in target acquisition scores for individuals who received the simulation 
treatment and those who received the non-simulation comparison application while performing in an HRC setting. 

As illustrated in Table 1, results suggest that there was a statistically significant difference in target 
acquisition scores, (Fi n = 4.68, p = 0.045) based on type of intervention received. The simulation treatment (M 
= 127.42, SD = 109.26) had a higher mean score than the non-simulation comparison application (M = 80.25, SD = 
66.69). Approximately 22% of the variance in score can be accounted for by intervention. 

Table 1 

Findings Based on Intervention Received 

Simulation Non-Simulation 

DF F Sig Partial M SD M SD 

Eta Sq 

Intervention 1,17 4.679 0.045 0.216 127.42 109.26 80.25 66.69 



Hypothesis II 

There is no significant change in target acquisition scores from game 1 and game 2 based on practice 
while performing in an HRC setting. 
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As illustrated in Table 2, results suggest there was no statistically significant change in target acquisition 
scores (Fi n = 1.77, p = 0.20). Although game 2 scores (M = 70, SD = 69.55) were higher in mean than that of game 
1 (M = 45.57, SD = 49.48), only 9% of the variance in score can be accounted for by practice. 



Table 2 

Findings by Game 1 and Game 2 

Game 1 Game 2 

DF F Sig Partial M SD M SD 

Eta Sq 

Practice 1,17 1.769 0.201 0.094 45.57 49.48 70.00 69.55 



Hypothesis III 

There is no interaction between type of intervention received and practice in an HRC setting. 

As illustrated in Table 3, results suggest there was no significant interaction between intervention and 
practice (Fi 17 = .19, p = 0.67). Only 1% of the variance in score can be accounted for by intervention. 



Table 3 

Interaction Findings Based on Game and Intervention 

DF F Sig Partial 

Eta Sq 

Practice* 1,17 0.190 0.67 0.011 

Intervention 



Discussion 



Hypothesis I 

Eindings suggest a significant difference {p < .05) in the performance of novice players in the number of 
targets acquired in a RE based on type of training intervention received. There may be a number of explanations for 
this significance. 

Eirst, significance can be explained by the reduction in cognitive overload due to prior practice in a 
simulated environment (the simulated desktop computer-based game). As noted by Atkins and Norris (2004), stress 
plays a role in the creation of cognitive overload; however experience also plays a role in the improvement of 
scores. 



A second plausible explanation for significance is that the intervention in the present study used different 
cognitive applications for acquiring the basic procedures for target acquisition. The simulation treatment can be 
argued as falling under the mechanical cognitive process (i.e., using the mouse, navigating, and searching), whereas 
the non-simulation comparison application falls under the verbal based cognitive process (i.e., reading only). Those 
who received the simulation treatment may have performed better because the mechanical application was 
experiential and, therefore, easier to assimilate than the verbal application of text alone. To help determine which 
intervention had a greater effect, selected cases were conducted and studied. In such cases, the statistical 
significance for Group A{p < 0.002) and Group B (p < 0.003) were equal. Literature does support the use of 
learning tools. Eor example, in an experiment conducted by Munzer, Zimmer, Schwalm, Baus, and Aslan (2006) for 
wayfinding, a map-based condition was shown to be significantly better than an auditory, walking guided condition 
(Fi, 6 o = 19.54, MSE = 0.0135, p < 0.001). In a study by Foreman, Stanton, Wilson, & Duffy (2003), a virtual 
desktop computer display, used to aid in improved performance, resulted in a significant main effect, the magnitude 
of which was large (Fi 10 = 10.83, MSE = 1002.21, p < 0.01); indicating that those trained in the Foreman, et ah. 
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study using the VE, made fewer errors than those trained with a physical model. In the present study, either 
intervention could be seen as a learning tool but favored the VE condition over the physical model. 

Einally, because the evidence suggests there is a link between the intervention (learning) and performance 
(target score), it is assumed that some form of transfer took place. Eor those participants who did not score a hit, it is 
likewise assumed that negative transfer could have been experienced (Leberman et ah, 2006) in response to finding, 
aiming, and shooting. Eor those participants who received the treatment and were successful in scoring hits, a type 
of complex transfer (Leberman et ah, 2006) could have been experienced. This complex transfer involves the ability 
to translate participants’ skills from the act of using a keyboard to aim and fire in a VE to the act of using a marker 
to aiming and shooting in a RE. In addition, near and far transfer (Leberman et ah, 2006; McKeachie, 2001; Royer, 
1986) could have been achieved for those participants who were able to acquire a hit in the RE. Near transfer could 
have been experienced by participants who were already proficient at shooting a hand gun and were able to shoot a 
marker correctly in the RE. Ear transfer could have been experienced by participants who received the non- 
simulation comparison application and yet were able to find, aim, and shoot a marker correctly in the RE. Einally, 
significance may be due to the error in self-report of the number of actual targets hit. 

Hypothesis II 

Eindings suggest there was no significant change {p > .05) in target acquisition scores based on practice 
while performing in an HRC setting. Although the mean scores from game 1 to game 2 did increase, indicating an 
improvement, findings suggests there were no significant difference at an alpha level of 0.05 between the means. 
Lack of significance may be found in four potential areas: (a) the small sample size (n = 19); (b) the limitation in 
time with the interventions (two, 10-minute segments); (c) the limitations in time with the practice sessions (game 1 
lasted approximately 2 Vi minutes, whereas game 2 lasted approximately 5 minutes); and (d) in the length of the 
break period (one hour). 

If the participants were given more practice time with either of the interventions, in the number of live 
practice sessions played or in the number of sessions played during the break period (more opportunity to play 
against experienced players), target acquisition scores may have improved, over time, from session to session. Eor 
example, in two studies, one by Ackerman (1992) and another by Bebko, Demark, Im-Bolter, & MacKewn (2005), 
practice had a positive effect on learning. In over 19 practice trials, Atkins reported results confirming the 
significant effect of practice (ES194900 = 205.68, ps < 0.01). Practice significance was also shown by Bebko et al. 
(2005) in the within-factor (F535 = 20.26, p < 0.0001) and the between factor (Fi j = 6.58, p < 0.037) for juggling 
skills in over 26 practice sessions. 

Finally, because findings suggest there was no significance in target acquisition scores, it could be assumed 
that participants did not accomplish two things: (a) a higher level of cognitive processing, and (b) the associative 
phase in Ackerman’s (1988) theory. Higher level processing requires connections to be made between the 
environments and experience (Desse, 2001; McKeachie, 2001; Price & Driscoll, 1997) but, which novice players 
may be ill-equipped to achieve (Kuhn, Black, Keselman, & Kaplan, 2000). In addition, higher level processing is 
accomplished through communities of practice (Greeno, 1998); in which participants would have developed 
predictable patterns of behavior on the paintball field. To accomplish the associative phase, the basic procedures in 
finding, aiming, and shooting had to be learned. These basic skills were probably not formed at a significant level 
due to limited exposure to the interventions or experience in live game play. 

Hypothesis III 

Findings suggest that there was no statistically significant interaction (p > .05) between intervention 
received and practice. Practically, the target acquisition scores did improve, although not significantly from game 1 
to game 2, which may suggest a natural tendency of improved behavior with exposure to an activity. Due to the 
parallel movement in scores, it appears that practice did not have a greater or lesser effect on one type of 
intervention over the other. Had the present study included a mix of novice players with experienced players there 
may still not have been an interaction effect, as indicated in a juggling study conducted by Bebko et ah, (2005), in 
which no interaction was found (F5 35 = 1.72, p > 0.205). In the Bebko et al. (2005) study, the experienced jugglers 
remained ahead of the novice jugglers throughout the study. 
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Theoretical and Practical Implications 

Theoretically speaking, the results of the present study suggest a movement towards the attainment of 
higher critical thinking described in Ackerman’s (1988) theory. Some participant’s seemed to advance from the 
cognitive phase to the associate phase, while others seemed to remain in the cognitive phase. However, if 
participants are provided the opportunity to gain enough practice time, either through additional exposure to the 
type of intervention or through additional live practice sessions, participants would then be able to advance from the 
basics learned in the cognitive phase to the connection of patterns found in the associative phase. 

Practically speaking, the significant difference in target acquisition scores and the type of intervention 
received hold implications for the type of training applied to those learning skills for HRC settings through 
interaction with various communities of practice, including practitioners, instructional technologists, and 
researchers. For the practitioner, the advantages of using a desktop simulation, rather than text-based non-simulated 
material, should be considered for tasks that are physically demanding, dynamic in nature, and involve complexity 
and risk. Since it appears that various dimensions of transfer did take place, instructional technologists should 
consider the selection of an appropriate use of a desktop computer simulation as a type of medium to use for HRC 
training applications. Determination of the proper training methodologies to apply to specific HRC settings would 
be of great value to decrease the learning gap and increase safety while advancing students from basic knowledge to 
practical application. The research community should continue to explore and quantify desktop simulations as a 
training medium and explore the variables that would provide the greatest effectiveness of transfer of learning to 
HRC environments. For example, if earlier research suggests that simulation can provide a better method for 
learning how to strategize in an HRC setting, and simple textual material provides a better method of introduction to 
learning fundamentals in preparing for an HRC encounter, then a multi-media approach should be researched to 
define the specific dimensions of transfer. 

Limitations and Future Directions 

There are a number of limitations including statistical concerns and uncontrolled extraneous variables that 
impact the utility of the results presented here. First, the research design used in the present study was limited in its 
analysis because baseline data, typically gathered from a pretest, and prior to intervention, was not available. 

Instead, the raw scores of each participant were analyzed. A second limitation was the reliance on memory and 
recall from participants for the number of opponents hit while engaged in live practice sessions. Due to outdoor 
lighting conditions, video recordings proved difficult in confirming hits self-reported by the participants. Video 
recordings also proved difficult in confirming the time when each participant exited the field. This inability to 
observe all participants during live practice sessions and the lack of adequate video capture forced reliance on the 
participant’s verbal recall. According to Ackerman’s (1988) theory, a high cognitive demand would have occluded 
the participant from remembering a count when the basics of finding, aiming, and shooting were still the main 
focus. Consequently, the significance reported in the present study may be due to the error in self-reporting of actual 
targets by participants. A third limitation is the lack of data on a participant’s efficiency (amount of paintballs used 
to acquire a target). This efficiency rating would have helped to better quantify a participants’ target acquisition 
skill. Finally, the generalizability of the present study was heavily weighted toward male gender and the majority of 
participants showing a strong propensity for shooting capabilities based upon findings from the classification 
matrix. 



A variety of procedural limitations should be considered for those wishing to replicate or use findings from 
the present study. First, sample size and statistical analysis should be considered: (a) statistically, the number of 
subjects is of concern; the complete sample size ended at 19, whereas a robust study would have consisted of a 
minimum of 32 participants, (b) A pretest, to determine actual level of ability, should be considered to help 
overcome the low sample size, (c) Post-hoc tests were not used because of the design adopted. Adding a control 
group as a third level will provide additional findings and allow for post-hoc analysis, if required, (d) Time exposed 
to the simulation treatment was limited to two, 10 minute sessions. Increasing this duration would provide greater 
exposure to the VE and subsequently lead to a greater potential of transfer to the RE. 

Second, video and visual issues: (a) certain aspects of the video recording process contributed to the 
degradation of the quality of the visuals. A mesh protection of two of the cameras created a fuzzy appearance when 



163 




viewing the videos. Protecting the cameras with Plexiglas opposed to mesh would enhance the video quality. The 
visuals were affected by the outdoor lighting which could have been corrected by placement of the cameras inside 
the protective mesh area or by using a smaller, well lit indoor field. A fourth camera would have also provided extra 
coverage needed for such a large playing field and potential data regarding participants not viewable from the other 
three cameras, (b) Aside from the visual quality, being able to track a paintball in flight was not possible. The speed 
of the paintballs exiting the marker prevented video capture with commercially available video equipment. It was 
therefore difficult to determine if the participant applied a strategy, like selective shooting, or if a random spray 
technique was employed. 

Third, enhancements: (a) to enhance the non-simulated comparison application, a plastic pistol should be 
provided to participants reading about shooting fundamentals. This may provide non-gun owners a better sense of 
finding, aiming, and shooting even though the markers did not have sights for aiming, (b) Being able to manipulate 
specific variables within the training context would provide the opportunity to examine aspects of one medium to 
determine which variables have a greater impact on performance. In addition, adding in factors that influence 
teaming and strategy formation could aid in the transfer of learning, (c) Having a mix of experienced players within 
the teams would provide a more representative sample of typical HRC situations. For example, the introduction of 
novice players (which represent the community) mixed with experts (which, for example, represent experienced 
police officers or military personnel) may show an increase in skill for novice players based on tangible examples of 
the successful behavior, as well as provide the potential benefits of teaming and strategizing. 

In conclusion, the results of the present study suggest a movement towards the attainment of higher critical 
thinking described in Ackerman’s (1988) theory. Some participants seemed to advance from the cognitive phase to 
the associate phase for several of the reasons described earlier, while others seemed to remain in the cognitive 
phase. However, if participants are provided the opportunity to gain enough practice time, either through additional 
exposure to a simulated or non-simulated intervention or through additional live practice sessions, participants 
would then be able to advance from the basics learned in the cognitive phase to the connection of patterns found in 
the associative phase. As research and instructional approaches advance, improvements in the design or capabilities 
to aid in the transfer of learning will directly affect the handling and representation of life-threatening situations 
found in HRC settings. These improvements may not necessarily be aesthetic in nature, such as having higher 
quality images, nor may they necessarily require having fancier models, such as one-to-one replication. They may, 
however, promote the selection of the proper training alternative for HRC skills and knowledge that lead to 
improved critical thinking vital to HRC situations. 
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Abstract 

Distance education is an education model which answers to needs of the students, academicians and 
educational materials via communication technologies. The most important characteristic of distance education 
model is its flexibility. Flexibility not only eliminates the obligation for students and academicians to convene at 
the same place, but also provides an educational environment without any boundaries. Interaction is also a 
desired attribute that should be expanded in distance education systems. The increase interaction leads to more 
active learning experiences. The main distinction between traditional and distance education is the use of mass 
communication methods and tools in education. There are various communication methods and media that can 
be used in education and specifically in distance education environments. It goes without saying that the video 
conference systems are gaining popularity in distance education. 

Bilateral video communication is provided between individuals or groups in different locations via video 
conferences. In a other words, people in two or more different locations can communicate by seeing and hearing 
each others. This study seeks to find out effective use of video conference system as a distance education media. 

For achieving this, a literature review was conducted on the use of video conference systems in open 
education, and then students’ opinions on distance courses applied in Anadolu University via video conferencing 
and its technical characteristics were determined. Learning attitudes and the expected success levels of the 
students from the video conference applications is revealed by explanatory factor analysis (EFA) of data 
acquired by a survey. Afterwards, Structural Equation Model (SEM) and Confirmatory Eactor Analysis (CEA) 
were employed to determine other factors which can have effects on success-learning and success-expectation 
levels. A model was proposed on the basis of CFA findings and the model was unravelled to academics. 

Key words: Distance Education, Video Conference, Structural Equation Models 

Introduction 

In today’s global world, video conference systems, that enable synchronous communication between peers 
without necessitating mobility, are extensively used not only in military, commercial and medical applications, 
but also, in educational purposes, especially in distance education. Video conferencing not only saves time, 
which in the current climate is one of the most important assets for people, but also reduces travel costs. While 
participants express themselves to others in different locations via microphone and camera, they use televisions 
and computer monitors to capture displays, and speakers to ascertain or detect voices. Video conferencing 
enables people to communicate face-to-face in comfort through technological support. Furthermore, video 
conferencing also enables the users to use various presentation techniques (Peter, 1998, p 139-140). 

Video conference diminishes time losses resulting due to the travels involved for attending meetings and 
educational centers. In this way, it enables participants to designate meetings in a short time and accomplish 
rapid decisions. These results in increased productivity, communication links are strengthened and a feeling of 
belonging is created. At the same time, both organizational costs and location dependency are reduced due to 
decreasing mobility. Video conference enables high-attending meetings by minimizing time and costs problems 
and raising frequencies and cheaper meetings by reducing education costs. 

The connection required for video conferencing is realized by the use of telephone lines, ISDN, satellite 
(VS AT) and coaxial or fiber optic cables. Each connection style has different costs and characteristics. 
Connections from furnishes one point to another are usually named point to point connection. Connections from 
one point to multiple points are also possible but a multiple control unit (MCU) is required. 

Today, videoconferencing is one of the most rapidly developing information and communication 
technologies providing bilateral communication between students and academicians or lecturers who are in 
different geographic location. These systems support effective, flexible and interactive communication and 
teaching environments and also reduce time and transportation costs. 

Video conferencing as a supporter of distance education 

Thanks to the ability of offering different opportunities by using integrated communication technologies, the 
most essential functions of video conferencing can be listed as informing and teaching. By submitting 
synchronous interaction, video conferencing usually uses printed materials and other instruments which 
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addresses both students and lecturers as supporters in open education, generates the feeling of belonging among 
the students, introduce new training methods and reinforce the institutions image (Race, 1998, p 1 10). However, 
problematic issues have emerged in video conference applications in open education. These include finding 
available linkage time of students, high cost of systems and contents of presentation. Therefore, this education 
discipline is faced with new discards, problems and applications. Video conference applications should be used 
according to the characteristic of students and the expected learning outcomes of the educational programs. 

Video conference systems incorporate instruments composed of camera, microphone and codec which 
provide synchronous voice, media and data transfer on regular data links or ISDN links. These instruments may 
be found as integrated systems including camera, codec and microphone and can also be obtained separately and 
assembled together (Forstyth,1998, p, 29)In the videoconference applications (Peters, 1998, p 211; 
Forstyth,1998, p 125; Mason 1994, p 74-84): 

-Negotiation in the form of questions and answers: This is the most used application of videoconference 
technology. People who use videoconference systems can negotiate mutually on a subject in a classical way and 
exchange their ideas. When compared to teleconferencing the difference is that videoconference systems allow 
participants to see each others facial expressions and gestures. 

-Watching and recording cassettes from video during videoconference discourse: Videoconference enables 
people to show a video to their counterparts or record the negotiation by video. Thanks to this application, which 
is advocated in distance learning, exchange of views can be done synchronously. Also, in watching a video 
cassette it allows the other side to brainstorm at the same time. 

-Showing printed documents by videoconference systems: By using a second camera, printed documents can 
be shown to the other party. Project drawings, samples, experiment forms are shown and the necessary 
information is given. For example; the way of filling a job form can be shown to the attendees interactively who 
congregate in the education saloon. 

-Showing the application on PC: By the PC connection interface at the back of videoconference systems, the 
data in the integrated PC can be displayed on the screen. This enables the educator to show the programs to the 
students in real time. 

For the success of the videoconferencing applications the training of the educators is very important. The 
course materials should be prepared in an attractive way and presented by the use of presentation techniques. 
The educators should also be motivated about lecturing via videoconferencing, and they should be familiar with 
the technical aspects of the system since lecturing via videoconferencing poses some difficulties on the side of 
educators. 

Videoconferencing Applications used as a supporter of distance education in Anadolu University 

Anadolu University has been renovating its communication technology capabilities since 2005, and is now 
able to conduct lessons by the help of video conferencing center at Yunusemre Campus Eski§ehir to other 
centers. These centers are located in Eski^ehir, Ankara, Istanbul, Siirt, Lefkose in Northern Cyprus and Cologne 
in Germany (Self-Evaluation Report 2008: 27) All centers have the capability to make multi conferences and 
also, each center is able to communicate with each other. During the conferences, it is possible to save or record 
voices, PC applications and board applications, and it is also possible to transfer the latest records to the 
conference. Currently, all the established centers communicate via IP units due to their lower prices. This 
method made it possible to administer video conferences all around the world. 

Anadolu University is one of the leading universities in the use of novel communication technologies. The 
underlying purposes are to increase the effectiveness of teaching facilities, and also offers these newer 
technologies to the benefits of its students. Initially, teaching materials such as printed ones, TV education 
programs and academic consulting were being used whereas now the unlimited capabilities of internet 
technologies have started to be used. Although limited, video conferencing systems are being used for academic 
counselling. Videoconferencing facilities in Anadolu University started in 1997 by the use of continental data 
lines and experimental satellite applications. In 1998-1999 academic year, videoconferences supported teaching 
experiments which were generated between Anadolu University and Ahmet Yesevi University in Turkistan, 
Kazakhstan. In the context of a pilot project marketing courses in the Faculty of Economics in Ahmet Yesevi 
University is done by the support of such videoconferencing. By the development of technical infrastructure, 
academic counseling for distance education students of Anadolu University in Lefkosa, Northern Cyprus, was 
introduced in 1999-2002. Academic counselling has been made available for the students in Northern Cyprus 
since 2003-2004 academic years. In like manner, 22 hours of videoconferencing weekly is being provided for 16 
courses via the contribution of 15 academic counsellors for the students in Siirt. This system works on IP based 
internet technologies. This application enables educators in Eskisehir and students from Eskisehir, Northern 
Cyprus and Siirt to participate in classes in an interactive manner. Notes prepared for the courses being held via 
videoconferencing are sent to the students one day prior to the session. As such, students have prior knowledge 
about what is going to be taught in the videoconference session 
( www.anadolu.edu.tr/aos tanitim/ogrenim ortamlari.aspx , 28.10.2008) 
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Problem 

Thanks to the latest technological developments, Anadolu University Open Education Faculty provides 
academic counselling services to students by video conferences. However, the contribution of basic factors such 
as content, technological infrastructure and teaching style to the students’ learning and desired success levels is 
not yet known. 



The Objective 

This study aims to determine the usage specifications of videoconference in Anadolu University and how this 
service can be enhanced and how students evaluate the factors which affect the implementation of DERA. This 
study also proposes a new model for a better quality videoconference course service. 

Limitations 

The study is limited to the views of students from Eskisehir, Siirt and Northern Cyprus who study in Distance 
Learning schools of Anadolu University and attended video conference courses in May 2008 

Methodology 

The oldest and best known statistical procedure for investigating relation between sets observed and latent 
variables is that of factor analysis. (Byrne, 2001, P 5). Factor analysis is a modeling approach that was first 
developed by psychologist as a method to study unobservable, hypothetically existing variables such as 
intelligence, motivation, ability, attitude, and opinion. Latent variables typically represent not directly 
measurable dimensions that are of substantive interest to social and behavioural scientists. (Raykov & 
Marcoulides, 2006, p 116). There are two types of factor analyses: exploratory factor analysis (EFA) and 
confirmatory factor analysis (CFA). (Byrne, 2001, p 5). 

Confirmation is accomplished by comparing the computed covariance matrix implied by the hypothesized 
model to the actual covariance matrix derived from the empirical data. Covariance structural analysis (CSA) 
utilizes the covariance matrix rather than individual observation as input, covariance structural modeling is an 
aggregate methodology. CSA unlike regression and ANOVA, individual cases or observations cannot be 
represented /predicted. (Diamantopoulos & Siguav, 2000, p 5). 

CFA is not considered with discovering a factor structure. CFA is considered to be a general modeling 
approach that is designed to test hypotheses about a factor structure, when the factor number and interpretation 
in terms of indicators are given in advance. Structural equation modeling (SEM) can be used for CFA. ( Raykov 
& Marcoulides, 2006, p 117). SEM is a statistical methodology that takes a confirmatory approach of a 
structural theory bearing on some phenomenon. (Byrne, 2001, p 4). 

SEM are widely employed in many scientific fields is that they provide a mechanism for explicitly taking in 
to account measurement error in the observed variables in a given model. In addition to handling measurement 
error, SEM also enables researchers to readily develop, estimate, and complex multivariable models, as well as 
to study both direct and indirect effects of variables involved in a given model. The combination of direct and 
indirect effects makes up the total effects of an explanatory variable on dependent variables. (Raykov and 
Marcoulides, 2006, p 7).Bolen and Long (1993) describe five stages characteristics of most applications of SEM: 
Model specification, identification, estimation, testing fit and respecification (Kelloway, 1998, p 7). 

SEM is very much a large sample technique. Both the estimation methods and test of model fit are based on 
the assumption of large sample. It is commonly recommended that models incorporating latent variables require 
at least a sample size of 100 observations, although parameter estimates may be inaccurate in sample of less than 
200 observations. (Kelloway, 1998, p 20). 

One of the easiest ways to communicate a structural equation model is to draw a diagram of it, referred to as 
path diagram, using special graphical notation. SEM is schematically portrayed using particular configuration of 
four geometric symbols: a circle, a square, a single-headed arrow and a double-headed arrow. (Byrne, 2001, p 
8).Three important assumption underlines path diagrams. First, it is assumed that all of the proposed causal 
relations are linear. Second, path diagrams are assumed to represent all the causal relation between the variables. 
Third, path diagrams are based on the assumption of causal closure; this is the assumption that all causes of the 
variables in the model are represented in the model. (Kelloway, 1998, p 10). 

The findings of students’ departments, courses and attendance frequencies were presented as descriptive 
statistics. Factor analysis was conducted in order to determine the factors affecting the implementation of 
videoconference courses and this yielded 15 items. Then by using these factors, confirmatory factor analysis was 
made in order to determine the relationship between learning and success attitudes of videoconference students. 
Structural Equation Modeling was also used in confirmatory factor analysis and a model was proposed with the 
Lisrel 8.53 Program. 
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Questionnaire 

The views of videoconference students related to the videoconference courses, students’ departments, and 
satisfaction levels from videoconference courses were questioned with the questionnaire consisting close-ended 
questions. The questionnaire was delivered within the centers where videoconference courses were conducted 
before the final exam, after the results of the midterm exams were announced in May 2008. 

Face to face interview 

Students in North Cyprus were intensely interviewed between 15th and 16th May 2008 in Cyprus. 

Population and sample 

The population is the students who attend videoconference courses in Eski§ehir, Siirt and Northern 
Cyprus. 122 students who participated in the study forms the sample of the study. 

Findings and implications 

Table 1 depicts the demographic of the participants. When the departments of the students are considered, it 
is seen that 54.9% (n=67) of the sample belongs to the Faculty of Economics, 31.1% (n=) from Faculty of 
Business Administration, and 13.9% (n=17) from Open Education Faculty. 47.5% of the participants continue to 
the first grade, 29 . 5 % to the second grade and 23.3% to the third grade. 64.8% responded to the survey in 
Eski^ehir, 25.4% in Northern Cyprus and 9.8% in Siirt. 



Table 1: Demographics of the Participants 







Freque 

ncy 


Per 

cent 


Cumulative 

Percent 


Faculty 


Open Education Faculty 


17 


13.9 


13.9 


Faculty of Economics 


67 


54.9 


68.9 


Faculty of Business 
Administration 


38 


31.1 


100.0 


Grade 


First Grade 


58 


47.5 


47.5 


Second Grade 


36 


29.5 


77.0 


Third Grade 


28 


23.0 


100.0 




Eski^ehir 


79 


64.8 


64.8 


Locatio 
_D 


Northern Cyprus 


31 


25.4 


90.2 


Siirt 


12 


9.8 


100.0 



Videoconference participation frequencies of the students are shown in Table 2. According to table 2, the 
students’ attendance to Introduction to Economics is the highest (95.3%). This is followed by Public Finance 
(85.7%) and General Accounting (78 %). These findings indicate that students regularly attend courses 
conducted via videoconference. 
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Table 2: Videoconference Courses and Students’ Participation Frequencies 



Courses 


Participating 

occasionally 


Participated 
several times and 
stopped 


Participa 
ting on a 
regular basis 


Participa 
ting just 
before 
exams 


Cost Accounting 


7.1% 


21.4% 


64.3% 


7.1% 


Financial Management 


14.3% 


7.1% 


64.3% 


14.3% 


Introduction to Behavioural 


11.1% 


15.6% 


71.1% 


2.2% 


Sciences 


Introduction to Mathematics 


15.2% 


13.0% 


69.6% 


2.2% 


Occupation and Social Security 
Law 


8.3% 


91.7% 


.0% 


.0% 


Turkish Tax System 


13.6% 


86.4% 


.0% 


.0% 


Management Information 
System 


38.5% 


7.7% 


46.2% 


7.7% 


Introduction to Management 


11.1% 


6.7% 


75.6% 


6.7% 


Introduction to Economics 


3.1% 


1.6% 


95.3% 




Public Finance 


11.4% 


2.9% 


85.7% 




Statictics 


11.8% 


11.8% 


76.5% 




Introduction to Law 


10.2% 


12.2% 


75.5% 




Introduction to Accounting 


8.0% 


10.0% 


78.0% 




Theory of Economics 


11.8% 


85.3% 


2.9% 




Accounting Practices 


17.2% 


13.8% 


69.0% 




Marketing Management 


16.7% 


8.3% 


75.0% 





Table 3 depicts the findings of general satisfaction levels of participants related to the video conference 
courses. When table 3 is examined, it is seen that the ratio of highly dissatisfied students is zero (0). When other 
satisfaction levels are analyzed it is detected that 2.5 % of the sample is dissatisfied, 12.3% is somehow satisfied, 
45.9% is satisfied and 39.3% is highly satisfied. Regarding these findings, the majority of students are satisfied 
with the video conference courses. 



Table 3: Satisfaction Levels of the Students Regarding Videoconference Courses 



Level of Satisfied 


Erequency 


Percent 


Cumulative Percent 


Dissatisfied 


3 


2.5 


2.5 


Slightly satisfied 


15 


12.3 


14.8 


Satisfied 


56 


45.9 


60.7 


Highly satisfied 


48 


39.3 


100.0 


Total 


122 


100.0 





Table 4 shows the findings of students’ assessment of the environment and the media used for 
videoconference courses. According to the findings, 75.4 % of the students think that both the audio quality and 
visual quality are adequate. 73 % of the students find the monitor size adequate. However, 42.6 % of the 
participants state that the number of courses is inadequate while 32.8 % admit that the duration of courses is 
inadequate. Nearly one fifth of the students consider that timing of the courses is inadequate. These results 
reflect that students expect that course duration should be lengthened /expanded. When other assessments 
towards video conference courses are examined, it is found that 4.9 % of the students considered the content of 
video conference course is inadequate, while 51.6 % think that it is adequate. 64.8 % of the students evaluate the 
physical videoconferencing environment as being adequate. 
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Table 4: Students’ Assessments of the Videoconference Environment and the Media 





inadequate 


somehow 

adequate 


adequate 


Audio quality 


1 


.8 

% 


29 


23 

.8% 


92 


75. 

4% 


Visual quality 


3 


2.5 

% 


27 


22 

.1% 


92 


75. 

4% 


Number of 
courses 


52 


42. 

6% 


37 


30 

.3% 


33 


27. 

0% 


Duration of 
courses 


40 


32. 

8% 


36 


29 

.5% 


46 


37. 

7% 


Timing of the 
courses 


24 


19. 

7% 


43 


35 

.2% 


55 


45. 

1% 


Content 


6 


4.9 

% 


53 


43 

.4% 


63 


51. 

6% 


Physical 

environment 


7 


5.7 

% 


36 


29 

.5% 


79 


64. 

8% 


Monitor size 


9 


7.4 

% 


24 


19 

.7% 


89 


73. 

0% 



The statements which represent students’ attitudes towards videoconference courses are presented in table 5. 
The results indicate that the majority of the students mostly agree with the statements questioned. However, 21.3 
% of the students expressed their disagreement regarding the number of questions solved during the video 
conference courses. This result can signify that students expect more questions to be solved to prepare 
themselves better for their exams. Besides, 37.7% of the students disagree with the idea of using video 
conference applications in all courses of distance education, 26.2% of the students disagree with the idea of 
“there is no difference between videoconference courses and face to face courses in terms of comprehension”, 
and finally, 32 % state that there are interaction problems between students and lecturers. 



Table 5: Degrees of Agreement with the Statements on the Application of Videoconference Courses 



How much do you agree or disagree with each 
statement? 


Completely 

Disagree 


Disagree 


Slightly 

Agree 


Agree 


Completely 

Agree 


Course presentations make the learning process easier. 


0.0% 


2.5% 


8.2% 


30.3% 


59.0% 


Questions solved in courses consolidate the contents. 


1.6% 


1.6% 


1.6% 


32.8% 


62.3% 


Questions solved in courses are enough for students’ 
understanding. 


4.1% 


17.2% 


13.1% 


39.3% 


26.2% 


Using of this distance education technique in all 
courses is enough for learning. 


9.8% 


19.7% 


15.6% 


31.1% 


23.8% 


Using visual elements in courses (PPT presentation) 
support the contents. 


0.8% 


2.5% 


7.4% 


42.6% 


46.7% 


Course contents are associated with current topics and 
discussion platforms are obtained for students. 


4.9% 


11.5% 


22.1% 


38.5% 


23.0% 


There is no problem about the interaction between 
students and lecturers in courses. 


3.3% 


5.7% 


6.6% 


35.2% 


49.2% 


Continuous videoconference applications effect the 
students’ success in a positive way. 


2.5% 


1.6% 


3.3% 


32.8% 


59.8% 


1 can instantly ask about the issues that 1 don’t 
understand. 


4.1% 


13.9% 


12.3% 


32.8% 


36.9% 


1 don’t need any other support for the courses via 
videoconference. 


11.5% 


26.2% 


25.4% 


17.2% 


19.7% 


My motivation to study increased through 
videoconference courses. 


4.9% 


7.4% 


12.3% 


40.2% 


35.2% 


It is not disadvantages to follow an instructor through a 
monitor in terms of understanding the course content 


8.2% 


6.6% 


9.8% 


39.3% 


36.1% 
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I believe that videoconference courses will have an 
important contribution to my success. 


2.5% 


4.1% 


9.0% 


35.2% 


49.2% 


There is no difference between videoconference 
courses and face to face courses in terms of 
comprehension. 


9.8% 


16.4% 


9.0% 


24.6% 


40.2% 


There is no difference between videoconference 
courses and face to face courses in terms of student- 
lecturer interaction. 


10.7% 


21.3% 


9.0% 


26.2% 


32.8% 



Students’ attitudes which includes 19 items towards video conference applications were assessed on five 
point Likert scale format (strongly disagree to strongly agree) and the results are summarized in Table 6. After 
factor and reliability analysis, 4 items were excluded from the list of statements. Principal component factor 
analysis with varimax rotation for 15 items was performed for students’ attitudes towards video conference 
applications. After the reliability analysis of 15 items’ the Cronbach's Alpha value was found to be .886. It 
yielded a 4 factor solution with Eigen values greater than one. Examination of item statistics identified all items 
in scale that qualified the alpha level. The first factor with 5 item account for 42.055 % of the total variance and 
the second factor with 6 item account for 9.163% of the total variance. The third factor with 3 item account for 
7.836 % of the total variance and the fourth factor with 2 item account for 7.453% of the total variance. Because 
there is more than 0,50 % correlation between components, varimax rotation was performed. The 
appropriateness of factor analysis was determined by Barlett’s test of sphericity =1861.555, p=0,0001<0.01 and 
the test KMO = 0.858, Sig 0.000 (df=105). These statistics indicated that data is suitable for factor analysis. 

Table 6: Factor Components Yielded by Factor Analysis Related to Scale Maderine 



(Rotated Component Matrix (varimax)) 





o 

o 

3 


ponent 


Variance 

Explained 


1 


2 


3 


4 


B 


S19. There is no difference between 
videoconference courses and face to face 
courses in terms of student-lecturer 
interaction. 


,829 








42,055 % 


S17. There is no difference between 
videoconference courses and face to face 
courses in terms of comprehension. 


,730 








S8. There is no problem about the interaction 
between students and lecturers in courses. 


,727 








S13. 1 don’t need any other support for the 
courses via videoconference. 


,622 








S4. Using of this distance education 
technique in all courses is enough for 
learning. 


,574 








O 


S9. Continuous videoconference applications 
affect the students’ success in a positive way. 




,800 






9,163 % 


S14. My motivation to study increased 
through videoconference courses. 




,755 






S16. 1 believe that videoconference courses 
will have an important contribution to my 
success 




,721 






SIO. I can instantly ask about the issues that I 
don’t understand. 




,613 






S15. It is not disadvantages to follow an 
instructor through a monitor in terms of 
understanding the course content 




,568 






I 


S2. Questions solved in courses are 
consolidate the contents 






,829 




7,836 % 


S3. Questions solved in courses are enough 
for students’ understanding. 






,659 




SI. Course presentations make the learning 
process easier. 






,529 
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s 


S6. Course contents are associated with 
current topics and discussion platforms are 
obtained for students. 








,811 


7,453 % 


S5. Using visual elements in courses (PPT 
presentation) support the contents. 








,701 




Cronbach's Alpha (,886) 


Total Variance Explained 


66,507 

% 



To find out the relationship between video conference applications and students’ learning and success 
expectations the following hypothesis are suggested: 

Hypothesis: 

HI: Students’ attitudes towards question solving and presentation in videoconference courses (I) positively 
affect students learning and success expectations (O). 

H2: Students’ attitudes towards lecturing in videoconference courses (S) positively affect students learning 
and success expectations (O). 

H3: Students’ positive attitudes towards videoconference course applications (B) positively affect students 
learning and success expectations (O) 




While I, S, B were identified as independent latent variable, O were operationalized dependent latent variable 
on the modeled constructs which can be seen in figure 1. With regard to analyze the relation between video 
conference applications and students’ learning and success expectations, structural equation modeling was 
performed with confirmatory factor analysis. The proposed model is demonstrated in figure 2. 

When the observance measures of the model shown in Figure 2, it is observed that the model can be accepted 
for some observance measures, and for the rest is closer to the acceptable values. As such, it is possible to state 
that the model is acceptable. 



Model 


Acceptable 

Observance 


RMSEA = 0,091* 


0,05 <RMSEA<0,10 


SRMR = 0.070 


0,05 <RMSEA<0,10 


GFI = 0.84 


0,90 <GFI<0,95 


AGFI = 0.78 


0,85 <AGEI<0,90 


*(RMSEA p=0, 00081 < 0.05) 



For HI, the coefficient showing the relation between I-^ O is calculated as .4 land t value is calculated as 
2.80. This proves that there is a meaningful relation between I-> O. Moreover, as can be traced in the figure, the 
relation coefficient between SI in I component and I is calculated as 0.68. SI (Course presentations make the 
learning process easier.) is evaluated by the students as having a higher effect on students’ learning and success 
expectations. 

For H2, the coefficient showing the relation between S^ O is calculated as -.02 and t value is calculated as 
0.13. Since t =0, 13 < 2 the relation between S-^0 is not meaningful. 

For H3, the coefficient showing the relation between B-^ O is calculated as .55 and t value is calculated as 
5.07. Hence, we can trace a meaningful relation between B^ O. The relation coefficient between S19 in B 
component and B is calculated as 0.86. S19 (There is no difference between videoconference courses and face to 
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face courses in terms of student-lecturer interaction.) is evaluated by the students as having a higher effect on 
students’ learning and success expectations. 

S9 (Continuous videoconference applications effect the students’ success in a positive way) in latent 
dependent variable shown by O is found to be not meaningful according to the results of confirmatory factor 
analysis. 

The relation coefficient between dependent latent variable O and S9 is calculated to be 0.77. This shows that 
S9 have a higher effect on students’ learning and success expectations. 

Consequently, HI and H3 explained above were approved and the relation coefficients between I->0 and 
B^O were found to be meaningful. H2 was not commended and the relation coefficient between S ~^0 was not 
found to be substantive. The model is a proposal for identifying the factors affecting the students’ learning and 
success expectations and it can be further developed. 




Chi- Squar e=168 . 31, df=84, P-value=0 . 00000, RM3BA=0.091 
Figure 2: The Model Proposed for Videoconference Students’ Success and Learning Expectations 

Students’ Opinions About Applications 

“They want face to face courses with the lecturers who attend to videoconferences” . 

“They want courses especially related with quantitative issues instead of verbal courses and therefore they want 
to attend firstly theoretical lessons and after that they want to make exercises" . 

“Lecturers should use the new methods by pushing the lines of technology because of being able to teach 
better”. 

“Time schedule of the courses should be planned according to the needs of students and credits of the courses, 
and these courses should be done at evenings in weekdays and afternoons at weekends” . 

“The course materials of videoconference should be sent to the e-mails of students. Therefore a portal should be 
opened, so all course materials, PPT etc. can be loaded to the portal”. 
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“The announcement of videoconference applications should be done more effective”. 

“Videoconference applications provide to learn the information from its owner, its expert and its own place”. 
“Videoconference applications have positive effects on students’ success”. 

“The facilities of TV and Internet technologies can be used more effective during videoconference courses”. 
“Videoconference room should be very well designed in terms of light, heating, sound isolation and ergonomics. 
Students should be able to interact with each other”. 

“Videoconference applications help students to become socialized, help to improve their feelings of belonging, 
motivate them to study and increase the student-student and student-lecturer interaction ”. 

“The graphics and writings which are technically seen in the videoconference screen should be seen more 
effective ”. 

“The courses should be recorded and broadcasted via Internet”. 

“The communication between students and lecturers can not be technically provided on time, there can be some 
lateness. Ai a result, there can be difficulties about the comprehension and reinforcement of the courses and 
there can be some understanding problems between students and lecturers”. 

“There should be quantitative courses besides verbal courses. The dates and durations of the courses are not 
enough, 1 hour is not sufficient” . 

“Because of videoconference is something visual, it has positive effects on motivation and reinforcement of the 
topics. Because of the graphics and writings are small, there can be some difficulties about reading. It should be 
developed technically” . 

“There should be no exercises, but more expressions of courses (mathematics). Firstly theoretical lessons should 
be done and after that exercises should be given ”. 

“The lecturers should use different instructional methods. Courses should be given parallel with face to face 
ones and also they should be parallel to other instructional medium and interaction should be provided”. 

“There shouldn ’t be any changes in the course schedule related to cultural characteristics like religious 
holidays ”. 

“Videoconference applications should be given not only in afternoons but also different hours of the day. The 
courses should be given also in summer terms ”. 

“Videoconference applications provide to learn the information from its owner and its own place. It has a big 
effect on the students’ success, because the students who attended to the videoconference applications passed 
their courses”. 

“Videoconference room is insufficient in terms of light and heating. Also some of the lecturers are sufficient and 
some of them are not”. 

“The lecturer should be seen on screen during the whole course period”. 

“Videoconference applications help students to motivate each other”. 

Conclusions and Recommendations 

This study investigates learning attitudes and expected success levels of the distance education students of 
Anadolu University participating in videoconferences and the results shows the students’ views supported by the 
results of statistical analysis. The findings of the face to face interviews with the students and the statistical 
analysis of their responses are presented below: 

Students report that technical infrastructure (voice-video-physical environment) is sufficient, but the number 
of lessons and the duration of the courses are not sufficient for them. 

To determine the factors of video conference class implementation’s effects on the students’ learning and 
success expectations, a new model was offered according to the basis of structural equation model. When this 
model was examined, academician-student interaction is found to be effective on the students’ learning and 
success. 

In the videoconference course, students expect much more problems to be solved. This can be explained by 
the use of multiple choice questions in the exams (held three times a year) for evaluating the academic success of 
the students. This is mainly due to students’ focus on achieving the exams grades rather than learning the courses 
effectively. 

Academician-student interaction has a great effect on positive attitudes of student’s about the video 
conference classes. During the lecture, academicians’ practical explanations for the purpose of explaining theory 
and launching a discussion board are not evaluated as being effective. Also, students think that the presentation 
of lessons by the use of Power Point does not affect their success. This may be explained by the lecturers’ over- 
dependence on the Power Point presentations. Besides, such a medium breaks the interaction between the 
academician and students. 

As a conclusion, the lecturing style and the physical environment of videoconference courses used in distance 
education programs is critically important. In video conference systems, technological infrastructure is necessary 
for the quality of the courses. However, the use of intensive technology in videoconference courses is not enough 
in itself to increase the levels of students’ learning and satisfaction. 



176 




Students’ satisfaction from the videoconference courses and the level of participation rests on student- 
academician interaction as well as the quality of service offered. It may also be concluded that the content of the 
course, teaching style and the communicational attitudes of the lecturer have an important impact on the 
satisfaction of students and therefore is effective over the students’ learning and success. 
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Abstract 

Knowledge Management has gained attention in the past decades. However, recent theories of knowledge 
management lack the individual factors of knowledge construction. On the basis of the theories of distri- 
buted cognition and mental models, we develop and introduce a new approach of knowledge manage- 
ment. It takes the individual and the organizational aspects of knowledge into consideration. The main 
scope is to support organizational modeling and to make organizational knowledge transparent. There- 
fore, new technologies are needed. A pilot study on extemalization methods was conducted. In this study 
we implemented a possible extemalization instmment called T-MITOCAR. The results show that T- 
MITOCAR can serve as a basis for developing assessment instmments which support organizational 
modeling in a new and economic way. 

Keywords: Knowledge Management, Mental Models, Organizational Knowledge, Expertise, Distributed 
Cognition 



Introduction 

In the past decades, knowledge management has gained a lot of attention (Angus, Patel, & Harty, 1998). In 
the economic sector, knowledge and learning have become more important due to the reduced importance of 
"hard" resources like capital. In order to be able to compete with other enterprises, companies must constmct 
individual and innovative knowledge. Therefore, new technology solutions have been developed. 

New knowledge management approaches have also been generated in the domain of human resources 
(Nonaka & Takeuchi, 1997; Probst, Romhardt, & Raub, 1998; Senge, 1990; Argyris & Schdn, 2002). Nonaka 
and Takeuchi (1997) developed a knowledge management framework which is designed to support knowledge 
dissemination. The framework is designed to disseminate knowledge through the whole organization in five 
phases. Probst et al also generated a model based on common management circles. Three milestones were sig- 
nificant; goal setting, implementation, control. The model was developed to implement knowledge management 
in organization (Probst, Romhardt, & Raub, 1998). 

What all of the approaches which have been developed to date have in common is that they equate know- 
ledge with information (e.g. Probst, Romhardt, & Raub, 1998). Knowledge is understood as something objective 
which can be acquired. However, knowledge is individual and is constructed by a cognitive system itself 
(Strube, 2001). In addition, these models only concentrate on the organizational part of knowledge management 
and ignore cognitive aspects of constructing and sharing knowledge. Furthermore, these models do not give 
solutions to bridge the expert-novice gap. Experts and novices differ in the grade of expertise, skills, and know- 
ledge organization they possess (cf. Gruber, 1994; Pirnay-Dummer, 2006). Thus, information that is provided by 
an expert is not inevitably understandable for a novice. In order to disseminate knowledge it is important for 
adequate models to be provided to each member of the organization (Hinds & Pfeffer, 2003). Additionally, 
modern models are more like implementation plans for knowledge management. They do not provide concrete 
theories about knowledge management. 

In order to overcome these problems it is of utmost importance to generate a new integrated theory which 
focuses on the cognitive aspects of knowledge construction. We consider Model Based Knowledge Manage- 
ment to be an innovative approach for this task. 

Towards a Theory of Model-Based Knowledge Management 

According to the "distributed cognition" theory (Hutchins, 2000) an organization can also be seen as a cogni- 
tive system. Cognition is not limited to a single person; it can also be distributed among a social group and its 
environment. Cognitive processes require shared knowledge representation. This knowledge is also distributed 
among the social group and its environment. Documents in knowledge management systems are primarily just 
information. They become knowledge if a cognitive system is using this information to solve a task (Strube, 
2001 ). 

If an organization can be described as a cognitive system, knowledge management can be seen as a meta- 
cognitive subsystem. Its task is to monitor, control, and evaluate knowledge and knowledge construction. If the 
organizational knowledge is sufficient to solve a problem, the organization can simply recall a scheme. If the 
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organization does not have enough information, it must construct an organizational model ad hoc by inferring 
and combining the current organizational knowledge. This organizational model is not stable since the same 
holds true for mental models (Seel, 1991) because the organizational model changes depending on the know- 
ledge base. Additionally, an organizational model is only subjectively plausible, i.e. it is not inevitably "right" in 
an objective manner. 

Therefore, the main goal of knowledge management is for organizations to construct valid organizational 
models, because right decisions which are based on these models - whether they are assessed or not - are of 
utmost importance for economic success. In order to support organizational modeling, the organization's indi- 
viduals must be supported while they construct knowledge and mental models. 

• Individuals must be supported while they construct a mental model in order to ensure that they con- 
struct valid ones. Therefore, the preconceptions of the individuals must be taken into consideration. 

• Tacit knowledge must be prevented. Therefore, knowledge management must be problem based in or- 
der to ensure that the information is applied. 

• The main goal is for the organization's expertise to be made transparent to each member. Therefore, 
appropriate externalization methods must be used in order for the actual knowledge to be made transpa- 
rent. 

For organizational models, we propose a three layer generic model of knowledge representation. 




Figure 1 : Three Layers of Knowledge Representation 



The first layer represents stable schemata within the individuals, including all of their trained routines and 
stable knowledge. This layer also corresponds to the organizational routines and all other matters that can be 
successfully planned in advance. The second layer represents the mental models of the individuals, which react 
to unexpected changes in the proximity of their working environment. The models are built on the fly and are 
subject to rapid change depending on the task or situation. The second level should react locally in the individu- 
al’s field of influence where direct feedback from the environment can be derived. The third layer is the organi- 
zational model. Usually this model is implicit and - as mentioned - not visible or available at all. If an organiza- 
tion wants to assess its model, it has to implement fast assessment methodologies which do not require know- 
ledge to be assessed individually. Its methodology necessarily does not leave the compilation of the representa- 
tion of the organizational knowledge to the subjective interpretation of (again) individuals. 

Assumptions 

Based on the theoretical foundation from research on expertise and on the practical goals of knowledge man- 
agement, the following core assumptions for the unique organizational knowledge model arise: 

1 . Co-worker and expert models deviate from the organizational model. 

2. The mental models of people who form an organization deviate from each other if they are assessed in- 
dividually. Some of the model features complement each other even if they are in a specified and given 
subject domain. 

3. The set union of all individual models deviates from the organizational model 
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Method 



In our first study* on model based knowledge management, we assessed a knowledge base (derived from a 
text) and individual experts who constructed concepts maps which referred specifically to the text. Instead of 
using a whole organizational model we took a locally precise model (on a single text), which helped to control 
the conditions properly for experimental quality. We addressed the three assumptions made above with the fol- 
lowing hypotheses: 

1 . Individual expert models differ from the model derived from the text 

2. The experts’ concept maps differ from each other (although they are all about a given text from the 
experts’ domain). 

3. The set union of the expert models differs from the text model. 

To make sure that the text model is a representation which fits the experts understanding and that the differ- 
ences stated by the hypotheses above are not accidentally generated by non-valid instruments, a fourth hypothe- 
sis was tested in order to add control to the instruments’ quality. It was tested with a small questionnaire which 
asked how well the text model represented the text and how sure the experts were about this statement. Both 
answers were collected on a 5 point Likert scale. 

4. When confronted with the model derived from the text, the experts will agree that it is an admissi- 
ble representation of the subject domain. 

To assess the “organizational” model, which was represented in our case by a text rather than by many 
texts, we applied methodologies based on MITOCAR and T-MITOCAR (cf. Pirnay-Dummer, 2006, 2007) and 
the SMD Technology (cf. Ifenthaler, 2006, 2007). The methods for one or for many texts are the same within 
the applied methods. The individual representations are assessed by means of classical graphical concept map- 
ping. 

Whereas classical methodologies (e.g. Al-Diban, 2002) are used to let the learner (or expert) conceptualize 
their knowledge graphically, natural language oriented methodologies like MITOCAR use multiple phases from 
text to graph. The instrument is a computer-based instrument for knowledge and model assessment. The instru- 
ment MITOCAR (Model Inspection Trace of Concepts and Relations) is used to assess knowledge and mental 
models. MITOCAR can elicit, visualize, analyze, and compare the knowledge structures of groups within a 
specific subject domain on the basis of natural language. It can easily interface with other automated analysis 
tools, e.g. with the SMD Technology (Ifenthaler, 2006) or ACSMM (Johnson et. al 2006). All the subjects need 
to do is go through a two-phase web-based assessment procedure which takes approximately 1.5 hours for a 
group. Afterwards, MITOCAR generates automated reports which not only display the knowledge structure in a 
concept map-like format but also calculate and interpret several tests, e.g. multidimensional scaling and homo- 
geneity (within a group), and provide additional descriptive measures and graphs which help the subjects to find 
answers within the knowledge structure (cf. Pirnay-Dummer, 2006). It is methodologically of great importance 
for this study that MITOCAR led to homogeneous data repeatedly on eight subsequent studies on groups of 
experts, three of which contained the same field of expertise (and the same kind of experts) which we address in 
the present study. 



1 



The pilot study presented in this proposal will be followed by a full study which will be conducted this fall. 
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The identification mode is the first phase of MITOCAR and is a simple collection of statements on a given 
subject domain. Between the first and the second phase, a concept parser filters nouns (with and without at- 
tached adjectives) and builds a list of the most frequent concepts from the “mini-corpus”. The second phase 
contains the review, the construction, the verification, and the confrontation. In the review, every group member 
rates all expressions of the group for plausibility and for their relatedness to the subject domain. In the construc- 
tion, the subjects categorize concepts into groups which can be processed into model information using markov 
chains. Verification and confrontation are both modes for a pairwise comparison of concepts. Pairs of these 
concepts are rated by the subjects in the second phase of MITOCAR for their closeness and contrast. Addition- 
ally, the subjects rate how sure they are about their rating. The two modes differ only in the pairs of terms which 
get rated. In the verification mode subjects rate the terms which come from their own group (utilizing their own 
power of language), while in the confrontation mode they rate pairs from another group (typically from a group 
which they are compared to). This information is used to build (re-represent) the knowledge structure in the 
form of a concept map. The newly developed T-MITOCAR toolset aims at converting prose text in a compara- 
ble way directly to a graph like a concept map. 

T-MITOCAR stands for Text-MITOCAR and was derived from the original version of MITOCAR in or- 
der to enable syntactic heuristics to be used for the re-representation. 
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Figure 2: Sample graph generated by T-MITOCAR from a text on economics 



The weights ( 0 < W < 1 ) at the edges describe the overall weight for the whole noun-distance oriented ma- 
trix generated /ram the text. The weights within the brackets show the weights within the graph. This weight is 
also taken to generate the color of the edges. The strongest edge is red while the weakest (compared to the 
graph, not to the text matrix) is blue. The “text to graph” feature of the software is based on several parsing 
heuristics and can be used to assess the learners understanding by generating graphical information from his or 
her text as well as to generate conceptual graphs from texts which are used as learning materials. It may simply 
help to have the option of avoiding the effort of an expert model in everyday classroom settings, even if expert 
models turn out to work better than the automated representations. 

There are six core measures for the comparison of conceptual graphs coming from the SMD Technology (cf. 
Ifenthaler, 2006, 2007) and from MITOCAR (cf. Pirnay-Dummer, 2006, 2007). Some of the measures count 
specific features of a given graph. For a given pair of frequencies fj and f 2 , the similarity is generally derived by 

1 I/1-/2I 

S = 1 

which results in a measure of 0 — ^ , where s=0 is complete exclusion and s=l is identity. The other 

measures collect sets of properties from the graph (e.g. the vertices = concepts or the edges = relations). In this 
case, the Tversky similarity (cf. Tversky, 1977) applies for the given sets A and B: 

f(A n B) 

f(AnB) + a- f(A-B) + p- f(B-A) 

a and |3 are weights for the difference quantities which separate A and B. They are usually equal (a = |3 = 
0.5) when the sources of data are equal. However, they can be used to balance different sources systematically 
(e.g. comparing a learner model which is constructed within five minutes to an expert model which may be an 
illustration of the result of a conference or of a whole book). 

In order from a general surface-measure down to a propositional matching the measures are: 

Surface (SMD) 
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The surface measure (cf. Ifenthaler, 2007) compares the number of vertices within two graphs. It is a simple 
and easy way to calculate values for surface complexity. 

Graphical Matching (SMD) 

The graphical matching (cf. Ifenthaler, 2007) compares the diameters of the spanning trees of the graphs, 
which is an indicator for the range of the conceptual knowledge. It corresponds with structural matching as it is 
also a measure for complexity only. 

Concept Matching (MITOCAR) 

Concept Matching (cf. Pirnay-Dummer, 2006) compares the sets of concepts (vertices) within a graph to de- 
termine the use of terms. This measure is especially important for different groups which operate on the same 
domain (e.g. using the same textbook). It determines differences in language use between the models. 

Density of Vertices (MITOCAR) 

The density of vertices (cf. Pirnay-Dummer, 2006) describes the quotient of terms per vertex within a graph. 
Since both graphs which connect every term with each other term (everything with everything) and graphs 
which only connect pairs of terms can be considered weak models, a medium density is expected for most good 
working models. 

Structural Matching (MITOCAR) 

The structural matching (cf. Pirnay-Dummer, 2006) compares the structure of two graphs without regard to 
their content. This measure is necessary for all hypotheses which make assumptions about general features of 
structure (e.g. assumptions which state that expert knowledge is structured differently from novice knowledge). 

Propositional Matching (SMD) 

The propositional matching (cf. Ifenthaler, 2007) value compares only fully identical propositions between 
two graphs. It is a good measure for quantifying semantic similarity between two graphs. Not only is the use of 
terms tested (like in the concept matching), but also their interestedness into the knowledge structure. 

Findings 

The participants of the pilot study were experts (N=9) in the domain of Learning and Instructional Design 
from our Instructional Design and Educational Science program at the University of Freiburg. The study was 
conducted in two phases. In the first phase the experts were asked to construct a concept map for a certain text 
about project-based learning. In this phase the data for the first three hypotheses was collected. In the second 
phase the experts were confronted with a model constructed with t-mitocar. They were supposed to assess the 
appropriateness of the t-mitocar model as an expert model. The findings are presented only in a descriptive 
manner because we consider this study to be a pilot study in which we want to obtain initial findings which we 
can operate with. In the following, we present the individual hypotheses and our findings. 

Individual expert models differ from the model derived from the text 

Our first hypothesis was that the individual expert models would differ from the model derived from the 
text. The range of the measures is from 0-1, where 0 is complete exclusion and I is identity. The individual 
findings can be viewed in table 1 . 



ModNr 


Surface 


Gra.M. 


C. M. 


Struct. 


Dens. 


Prop. M. 






1 


0.36 


0.83 


0.18 


0.20 


0.56 


0.04 






2 


0.42 


0.86 


0.23 


0.11 


0.60 


0.04 






3 


0.97 


0.75 


0.33 


0.75 


0.75 


0.08 






4 


0.69 


0.86 


0.23 


0.43 


0.86 


0.00 






5 


0.69 


0.86 


0.23 


0.43 


0.86 


0.00 






6 


0.33 


0.67 


0.24 


0.06 


0.57 


0.04 






7 


0.72 


1.00 


0.23 


0.43 


0.61 


0.03 






8 


0.61 


1.00 


0.29 


0.71 


0.91 


0.07 






9 


0.75 


1.00 


0.18 


0.40 


0.64 


0.00 






Mean 


0.62 


0.87 


0.24 


0.39 


0.71 


0.03 


0.48 


Total Mean 


SD 


0.21 


0.12 


0.05 


0.24 


0.14 


0.03 


0.13 


Standard Error 



Table 1: Similarities (Individual Models x Text Model) 
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On the structural surface there was a high accordance between text model and expert model: The mean of the 
measures for grciphical fncitchillg (^'graphical matching ~ 0.87, sdgj-aphlcal matching “ 0.12), density (^'density “ 0.71, sdden.sity “ 
0.14), and surface (-tsurface= 0.62, sdsurface 0.21) were higher than expected. As regards the semantic measures, on 
the other hand, the expert models differed from the T-MITOCAR model. There was a low accordance in con- 
cept matching (^concept matching = 0.24, sdconcept matching = 0.05). Abovc all, the Semantic similarity measured by the 
propositional matching measure differed strongly from the text model. There was a mean of ^ propositional matching = 
0.03 with a standard error of sd propositional matching = 0.12. The total mean (mean of the individual similarity means) 
was ^totai = 0.48 with a standard error of sdtotai= 0.13. In general, the experts’ models were more similar to the 
text model than expected, but .48 is still too low to be considered close. 

The experts ’ concept maps differ from each other 

Our second hypothesis was that the experts’ concept maps would differ from each other although they were 
based upon the same text. The findings can be viewed in table 2. 



ModNr 


Surface 


Gra.M. 


C. M. 


Struct. 


Dens. 


Prop. M. 


1 X 2 


0.87 


0.71 


0.30 


0.56 


0.93 


0.00 


1 X 3 


0.35 


0.63 


0.25 


0.30 


0.75 


0.04 


1 x4 


0.52 


0.71 


0.30 


0.55 


0.48 


0.00 


1 X 5 


0.52 


0.71 


0.30 


0.55 


0.48 


0.00 


1 X 6 


0.92 


0.80 


0.16 


0.29 


0.99 


0.00 


1 x7 


0.50 


0.83 


0.11 


0.57 


0.91 


0.00 


1 x8 


0.59 


0.83 


0.38 


0.22 


0.51 


0.13 


1 x9 


0.48 


0.83 


0.17 


0.45 


0.88 


0.05 


2x3 


0.41 


0.88 


0.24 


0.17 


0.80 


0.00 


2x4 


0.60 


1.00 


0.21 


0.34 


0.52 


0.00 


2x5 


0.60 


1.00 


0.21 


0.34 


0.52 


0.00 


2x6 


0.80 


0.57 


0.23 


0.29 


0.94 


0.00 


2x7 


0.58 


0.86 


0.16 


0.34 


0.98 


0.00 


2x8 


0.68 


0.86 


0.22 


0.12 


0.55 


0.06 


2x9 


0.56 


0.86 


0.27 


0.33 


0.94 


0.10 


3x4 


0.68 


0.88 


0.24 


0.63 


0.65 


0.00 


3x5 


0.68 


0.88 


0.24 


0.63 


0.65 


0.00 


3x6 


0.32 


0.50 


0.31 


0.10 


0.75 


0.04 


3x7 


0.70 


0.75 


0.36 


0.63 


0.82 


0.03 


3x8 


0.59 


0.75 


0.35 


0.76 


0.68 


0.11 


3x9 


0.73 


0.75 


0.44 


0.51 


0.85 


0.13 


4x5 


1.00 


1.00 


1.00 


1.00 


1.00 


1.00 


4x6 


0.48 


0.57 


0.08 


0.14 


0.49 


0.00 


4x7 


0.96 


0.86 


0.27 


0.81 


0.53 


0.00 


4x8 


0.88 


0.86 


0.30 


0.47 


0.95 


0.05 


4x9 


0.93 


0.86 


0.22 


0.56 


0.55 


0.04 


5x6 


0.48 


0.57 


0.08 


0.14 


0.49 


0.00 


5x7 


0.96 


0.86 


0.27 


0.81 


0.53 


0.00 


5x8 


0.88 


0.86 


0.30 


0.47 


0.95 


0.05 


5x9 


0.93 


0.86 


0.22 


0.56 


0.55 


0.04 


6x7 


0.46 


0.67 


0.23 


0.18 


0.92 


0.11 


6x8 


0.55 


0.67 


0.32 


0.07 


0.51 


0.14 


6x9 


0.44 


0.67 


0.17 


0.20 


0.89 


0.05 


7x8 


0.85 


1.00 


0.33 


0.47 


0.56 


0.09 


7x9 


0.96 


1.00 


0.43 


0.68 


0.96 


0.04 


8x9 


0.81 


1.00 


0.28 


0.49 


0.58 


0.18 
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Mean 


0.67 


0.80 


0.28 


0.44 


0.72 


0.07 


0.50 


Mean 


SD 


0.20 


0.14 


0.15 


0.23 


0.19 


0.17 


0.18 


Standard Error 



Table 2: Similarities (Individual Models x Individual Models) 



There was a high accordance in graphical matching (^graphical matching = 0.8, sdgjaphicai matching = 0.14), density 
(^'density = 0.72; sdjensity = 0.19), and surface (^surface = 0.67, sdsurface = 0.20), but a low accordance in the rest of 
the similarity measures. The total accordance of the individual experts models had a total mean of iitotai= 0.5 
with a standard deviation of sdioiai = 0.18. The total mean was also higher than originally expected (also see 
table 2). 

The set union of the expert models differs from the text model 

Our third hypothesis was that the set union of all expert models would differ from the text model. The mean 
of the individual similarity measures was ^mtai = 0.33 with a standard deviation of sdtotai = 0.32. 



Surface 


Gra.M. 


C. M. 


Struct. 


Dens. 


Prop. M. 






0.24 


0.38 


0.29 


0.09 


0.93 


0.05 


0.33 


Mean 














0.32 


SD 



Table 2; Similarities (Union Set x Text Model) 



This finding favors our hypothesis, which states that the set union should differ from the text model because 
there is less accordance between text and expert model. A set union does not seem to represent an organization- 
al model. Therefore, other methods are needed to assess organizational models. 

When confronted with the model derived from the text, the experts will agree that it is an admissible repre- 
sentation of the subject domain 

After the first phase the experts were given a short questionnaire. They were supposed to rate whether the 
text model was appropriate as an expert model and how certain they were about their decision. The answers 
were collected on a 5 point Likert scale (1 = strongly disagree, 5 = strongly agree). In table 3, the main results 
are presented. 



Means Appropriateness 


3.0 


Standard Deviation 
Appropriateness 


1.05 






Means Certainty 


3.89 


Standard Deviation 
Certainty 


0.74 



Table 3: Assessment of the Experts 



The experts could not clearly decide whether the text model was appropriate as an expert model. There was a 
mean of ^appropriatene.ss = 3.0 with a standard deviation of sdappropriateness = 1.05. This mean is not supposed to be 
exactly in the middle, but as we can see the standard deviation is also very high. Most of the experts tend to say 
that the text model is also appropriate for use as an expert model. Just one expert judged the t-mitocar model to 
be insufficient as an expert model. The next item measured the certainty of their assessment. There was a mean 
of A'certaiiity = 3.89 with a Standard deviation of sdcenaimy = 0.74. The experts were quite sure about their decision. 
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Discussion 



Although differences were smaller than expected on the structural surface of the models, they seem to be 
semantically diverse. This can be observed throughout the hypotheses. We see that union sets of individual 
knowledge representations do not represent an organizational knowledge, nor do systematic subsets of it. There- 
fore, organizational knowledge must be assessed on the organizational level. This is of course not an easy task. 
Nor can we interview or directly ask the organization questions: classically, we can only assess what its people 
produce or observe peoples behavior. However, since the organization generates output on the basis of its know- 
ledge non-reactively, e.g. found in dossiers, notes, reports, and all kinds of written text documents, instruments 
which are language-oriented could compensate for the difficulties in assessment. Another important benefit 
which comes with this approach is that these kinds of assessments cannot be traced back to the individual - the 
data shows how independent the group model is from the individual conceptualizations. Thus, it could be easier 
to gain the necessary support for the assessment: It is the organization which is assessed. 

Furthermore, we must consider the practical task of knowledge management. It must support organizational 
modeling. Hence, the organizational model must be made transparent. It is of utmost importance for organiza- 
tion to have tools for model based knowledge management that can construct organizational models which can 
be used to plan economic decisions ad hoc on the basis of the output of the organization members. Therefore, 
applications like T-MITOCAR are designed to bridge the gap between accuracy and efficacy. Such tools can 
ensure an improvement in knowledge management because organizational knowledge can be made transparent 
to all of the organization members and organizational modeling can be enhanced. In addition, T-MITOCAR 
could also be used as a kind of navigational ontology within knowledge management systems because it abbre- 
viates the general characteristics of a database. This can help in the same way with automatic keywords, data 
mining, and smart archiving. In contrast, expert models or set unions of expert models have methodological 
weaknesses. Because they are artificial clusters without a common validation by the group, they can not be 
taken as comprehensive models of an organization without further shaping. Shaping them the traditional way 
takes a lot of effort and time. Also, the experts agreed that text models generated by T-MITOCAR are suffi- 
ciently appropriate as expert models. This provides a good base for further investigation. 

This study had the character of a pilot study. We will now test our hypotheses in a full study. Afterwards, 
our findings will be tested within a running knowledge management system of a company for validation and for 
evaluation of feasibility and practical dissemination. Preparations are already being made for a set of studies 
within knowledge based subject domains in companies. 
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Abstract 

Many teacher educators lack the skills necessary to model effective technology use in their 
university courses. The purpose of this study was to explore faculty perceptions of the role 
technology should and does play in the courses they teach. It also examined how they themselves 
use technology when teaching (specific examples), how they expect their students to use 
technology when teaching, and where in the teacher education program the use of technology as 
an instructional tool should or does occur. Using in depth interviews and qualitative research 
analyses, this study explores common themes in faculty perceptions across content areas (math, 
science, language arts, etc.) and grade levels (middle school and high school). 

Background Information 

Many states have adopted technology requirements for students preparing for the teaching profession which stress 
the use of technology in their teaching based on the International Society for Technology in Education National 
Educational Technology Standards (ISTE NETS). These goals include the implementation and integration of 
instructional technology to facilitate the teaching/learning process. Accordingly, most educators today believe that 
public schools must ensure the effective use of technology in the classroom in order to prepare our children for the 
21st century learning environment. In 2000, the National Center for Educational Statistics found that nearly all 
public school teachers reported having computers available for teachers somewhere in their schools. However, the 
increasing presence of technology in our nation's schools does not ensure its implementation. 

Erom preschools to universities, technology has become an integral tool in the teaching and learning process. Over 
the past decade, teacher education programs have sought ways to prepare pre-service teachers to effectively 
integrate technology into the classroom. Rather than rallying the support of the entire teacher education faculty, 
many programs have focused their efforts on a few faculty members with technology experience or a single 
technology course to address this need. Moreover, many teacher education students feel inadequately prepared for 
their role in today’s classroom. According to Carlson and Gooden (1999), teacher educator’s campus-wide must 
model technology use in order to prepare prospective teachers to integrate technology into their own instruction. 

In spite of the tremendous growth of technology in our nation’s schools and the belief by a majority of educators 
that all students must have access to technology to be truly successful in today’s world, there is evidence that many 
teachers still do not use technology at all in their teaching (Education Week, 1998). According to a national survey 
(DeMedio & Teclehaimanot, 2001), 90 percent of schools in ten states have Internet access in their classrooms. A 
report from the CEO Forum on Education and Technology (2000) found 80% of schools in the United States have 
access to technology, but few teachers are ready to use the power of technology in their classroom activities. A 
Department of Education Survey learned that only about 33% of teachers believe that they could use technology in 
their classrooms (National Center for Education Statistics, 2000). A national survey by Becker (1999) supported the 
finding that as many as 70% of teachers are not using the technologies available to them. 
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Technology Integration 



The lack of formal training in the use of technology as an instructional tool remains the most common barrier to the 
integration of technology in teaching (Zhao, Y., 2007). A pre-service teacher’s interest in the use of technology as 
well as practical experience using technology in the classroom have been found to correlate positively with the 
degree to which that pre-service teacher eventually integrates technology into instruction as a professional. Basic 
factors that influence teacher perceptions about technology use include support from the working community for the 
use of technology, a vision of how technology can support required curriculum, environmental support or the 
availability of technology, teacher interest and experience with the use of technology as a teaching tool, and a 
personal belief in the value of technology. 

A number of factors are critical to the success of a professional development program for university faculty. In order 
for faculty to integrate technology across the curriculum, there must be institutional support that rewards faculty 
with release-time, development grants, or other incentives. Furthermore, there must be a support infrastructure 
available to help faculty troubleshoot and solve technology related problems without delay and interruption of 
classroom activities. Colleges and universities must create technology training centers equipped with computers and 
peripheral devices that faculty can use to prepare course materials and to become proficient with innovative 
technologies. According to Maney (1999), it is not the technology that makes the difference, but rather how teachers 
adapt and apply technology. 

Faculty adoption of technology into the curriculum is key in transforming the teaching and learning process. Rogers 
(1995) stated that a positive attitude toward any innovation increases the likelihood of the adoption of the 
innovation. Faculty attitudes toward technology, fear factors and complexity issues, lack of time and support, limited 
access, inadequate faculty development opportunities, and lack of organizational support have all been identified as 
major barriers to the infusion of technology into teacher preparation programs. Many factors influence whether 
faculty will integrate technology into their courses including access to computers and Internet in the classroom, 
updated software and hardware, and release time for faculty to learn how to integrate technology into the curriculum 
(Maney, 1999). 

Time is a primary concern of faculty members and often provided as a reason for not using technology in the 
classroom (Strudler, McKinney, & Jones, 1995). Educators must be allowed adequate time to learn new 
technologies and experiment with new teaching strategies. 

While numerous studies have examined the use of technology in the K-12 classroom, few have explored the extent 
that technology is used and required in the undergraduate teacher education programs. The purpose of this study was 
to explore faculty perceptions of the role technology should and does play in the courses they teach. It also 
examined how they themselves use technology when teaching (specific examples), how they expect their students to 
use technology when teaching, and where in the teacher education program the use of technology as an instructional 
tool should or does occur. Using in depth interviews and qualitative research analyses, this study explores common 
themes in faculty perceptions across content areas (math, science, language arts, etc.) and grade levels (middle 
school and high school). 

This study evolved from a U.S. Department of Education Teacher Quality Enhancement grant that aims to recruit, 
train, and retain high quality math and science education students. One of the outcomes of the project is to “increase 
the number of teachers using educational technology themselves and with their students’’. Previous research which 
measured pre-service teacher use of technology during student teaching indicated that faculty members were not, 
according to the students, modeling the use of technology as an instructional tool. This exploratory study set out to 
discover where College of Education faculty members felt technology integration was being taught in the pre- 
service teacher program, how they themselves contributed to the modeling of technology as an instructional tool, the 
types of technology they felt with which their students should be competent, and their level of understanding or 
awareness of the ISTE NETS. 



Method 

The faculty sample, selected purposefully, consisted of those who taught Teaching Methods in the Department of 
Curriculum and Instruction at a medium-sized, Midwestern university. Eight faculty members were interviewed. 
Teacher Education programs included both middle and high school and several content areas: mathematics, science. 
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English/language arts, social studies and foreign languages. In addition, two college administrators were 
interviewed — one department chair and the Dean of Undergraduate Education. The interview protocol was based 
upon ISTE NETS standards for teachers in an effort to determine the degree to which they were aware of and 
incorporated those standards into their teaching. Interviews were recorded, transcribed and then explored for 
common themes. 

All teacher education majors are required to take a course in the use of technology as an instructional tool. This 
course. Technology & Multimedia in Educational Environments emphasizes the development of computing skills 
with a focus on productivity tools in organizing, managing, multimedia authoring, homepage development, software 
evaluation and presenting lessons for professional communication in K-12. This course provides students with 
learning experiences utilizing microcomputers and software applications and their use for information management 
and communications for professional productivity. It emphasizes developing the skills necessary to become 
proficient with technology and begin applying it to the teaching and learning process. Course goals include a 
general understanding of the capabilities and functions of microcomputer and related peripherals and accessing, 
organizing, managing, and presenting data for professional and personal communication and use. Topics include 
cross-platform computer operation, word processing, email, listservs, and WWW searching and resources, database, 
spreadsheets, and presentation software. During the three-hour semester course, students are expected to master 
basic microcomputer functions including common word processing, database, presentation, and graphic software; 
internet based tools including email, web browsing; multimedia instructional programming; html basics and the 
creation of a web page; and an understanding of various distance learning formats. There is no opportunity within 
this course, however, to apply what is learned to field experiences. In addition, the content is general and does not 
address content specific teaching situations. 



Findings 

Once data was collected and analyzed it was arranged under the common themes of why technology is taught to pre- 
service teachers; what type of technology is taught; where it is taught; when is it taught; and how is it taught. The 
faculty and administrators felt that the main reason technology proficiency is required within the College is because 
it allows the teacher candidates to “keep up” with what is new. Most faculty (5) and both administrators responded 
that there was a concern that the teacher candidates would graduate and get jobs and not know enough technology to 
make use of what might be available in their schools. Eour faculty and both administrators also noted in their 
responses that technology can enhance instruction. Two faculty and both administrators provided answers that 
illustrated a clear understanding of why NCATE requires the ISTE NETS standards — “to engage students and 
improve learning; enrich professional practice; and provide positive models for students, colleagues, and the 
community” (from ISTE NETS-T Indicators). A concern that our graduates would not be tech savvy was the more 
prevalent theme, however. 

When asked where and when pre-service teachers received training in the use of technology as an instructional tool, 
half of the faculty responded that it occurred in the college’s required technology course and four indicated that 
some training takes place in their Methods of Teaching course. Those that indicated they incorporate it into their 
Methods course also provided some examples of specific programs and tools in general and relating to their content 
area. Eour mentioned PowerPoint and Smartboard and five used the internet and online instructional tools like 
WebCT. In the content area, both math education professors cited technology tools like graphing calculators and 
Geometer’s Sketchpad. Of the three science education professors, only one (middle grades) mentioned content 
specific tools. He indicated that he introduces his students to the use of both probes and simulated laboratories 
(software) as part of the Methods course. The faculty from the other content areas did not have content specific tools 
but did mention using the internet to explore history, literature, and language. The two administrators both felt 
technology as an instructional tool was being taught in the Methods course, one administrator also mentioned the 
college’s technology course, and the other administrator indicated that other content courses taught on campus 
provided some exposure (like the use of “clickers” in chemistry courses). 

The faculty were also asked how frequently they themselves enhance instruction with technology. Eive of the eight 
faculty indicated that they do this every time they teach. One felt it occurred about 90% of the time, one indicated 
50% of the time and one confessed that technology was utilized only about 20% of the time. Tools they most 
frequently used were the Smartboard and a web-assisted program like WebCT. 
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One ISTE NETS standard is that teachers understand the process for appropriately integrating technology in order to 
enhance teaching and learning. When asked if they share this with their students, six said they did although only 
three provided some evidence beyond just talking about it. One faculty indicated that the process was learned 
through the development of a lesson plan and two (both math education) provided examples of situations where the 
students actually learned the process through hands-on activities. 

Yeah, we do have conversations about when it's appropriate to do different things. One of the things that we 
actually talk about a lot is how to use the space that you have, like your board, or your, when you're using a 
SMARTboard and how to help make things visual for students. So we, sometimes we'll stop and talk about is 
this really the best way to do this or not. But we do that really broadly across a lot of things that we, not just 
in terms of technology. But, like when we're using graphing software, we talk about where's the appropriate 
place. You know, where does this go in the curriculum. You know, where does it fit, and when's the best 
time to use it and how you might use it with one group of learners vs. another. (Math, middle grades) 

Well, by integrating the software. One, is by finding readily available software for them to use in school. 
GeoGebra is such software. Now, all they do is download it. They’ve had experience with it in other, in 
math classes, in geometry. So that they know how to use it and it isn’t a problem if the school, if it’s not 
available. It’s downloadable. It’s free on the Internet, or they can just go right to the web and use it. (Math, 
secondary school) 

Einally, one was rather surprised when asked about modeling the process stating, “I don't because I don't think of it 
as a thoughtful process. I've, I just think of it as, as a, just an opportunity, as just something you do. It's not 
something you necessarily planned for.” 

Our semester long student teaching field experience has a requirement that a lesson utilizing technology be designed 
but faculty members do not require that it be implemented. When asked if there is a requirement that technology be 
used as an instructional tool during student teaching, four faculty said no. Two weren’t sure and two others said that 
the lesson plan was the proof that they had accomplished the integration of technology as an instructional tool. 
Interestingly enough, both administrators responded with a resounding “yes” that this was indeed a requirement of 
student teaching. 

Two faculty members and both administrators pointed to the portfolios as evidence of the use of technology to 
enhance instruction. Student portfolios concentrated on one to two weeks of the student teaching field experience 
and focused upon one or more learning goals. The portfolio documented a series of lessons and showcased the 
environment the student teacher established for learning as well as the kinds of tasks that characterized the student’s 
teaching style. The portfolios included detailed lesson plans for at least five lessons, an analysis of the students’ 
learning, and reflection upon teaching. While the requirements for the student portfolio are “are aligned with the 
College’s conceptual framework, which outlines its vision in accordance with institutional and national standards”, 
there is no specific requirement that students illustrate through their portfolio their ability to use technology in 
teaching to enhance learning. The only required reference to technology is a list of technology available in the 
classroom that each student teacher must complete as part of the “Teaching Context Eorm”. There is no place on that 
form to indicate whether any of the technology available was utilized. 

Einally, faculty were asked to provide examples of how, within their subject area, teachers might use technology to 
facilitate active learning. Eour faculty (both math, one foreign language, and one science) provided examples that 
identified specific tools and situations where technology could be used. One science professor cited a rather general 
situation and the other science professor as well as the social studies professor cited specific examples that they 
themselves did not indicate they trained their students to use. Einally, the language arts professor provided an 
extended example but framed it within the following context: “. . .but I would have a problem with trying to demand 
that they use that [technology]. If you can do it better without the, you know, plugging anything in, well, I say more 
power to you, just go for it.” 



Conclusion 

Based upon our interviews, it is clear that the majority of the faculty do not consciously plan to teach students how 
to use technology as an instructional aid. Most responses indicated that it is used when they feel it is appropriate but 
they do not, in general, make a point of illustrating to students why and how to use technology when teaching. In 
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addition, most believe that technology training is important, or rather that the graduates are tech-savvy, so that the 
teacher candidates can compete with graduates from other universities as well as stay ahead of their students when 
it comes to understanding how to use technology. Based upon these interviews, it is believed that while the 
administrators are acutely aware of ISTE NETS standards, the faculty are not. Eurther research might examine 
whether an awareness of the ISTE NETS standards might improve the integration of technology as a teaching tool in 
the Methods courses. 
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Re-Evaluation of e-Transformation Efforts in Anadolu University Open 

and Distance Education System 

Cemil Ulukan 



Abstract: Anadolu University aims to create more student-centered e-learning environments so that the 
learning may be richer, more effective and more interactive. To this end, the university has in recent years 
provided digital versions of the printed distance course materials on the Web. Also, the Web-based student 
support services have been increasing and diversifying. However, it is suggested that the works undertaken in 
pursuit of what might be called ‘e-transformation’ are not being introduced and operated in a planned and 
systematic way, which lessens the impact of the changes and created a hectic work environment. To accurately 
evaluate the e-transformation efforts in the University, a broader approach towards e-transformation is needed. 
This paper analyzes and re-evaluates the e-transformation efforts in open and distance education of Anadolu 
University according to various dimensions. 



Introduction 

Most of world’s educational institutions are progressively building up experience in delivering courses using 
information and communications technologies (ICTs), either to enhance, extend or complement their in-class or 
off-campus education or create wholly online courses or programs. These developments have been accompanied 
by changes in the teaching and learning paradigm, and in some cases, constitute comprehensive e- 
transformation. This is especially true of the open and distance education institutions, and to a lesser extent some 
dual mode universities - because the advancements in ICTs have been seen as providing enormous opportunities 
for these institutions. Distance education, in all of its various forms, uses of technology and applications of 
blended learning, and is becoming more and more popular among learners since they provide convenient, 
flexible and effective alternatives to conventional studies, rich and collaborative learning environments and 
increased and enhanced interactivity. However, it is posited that the creation of above learning environments 
calls for administrative, pedagogical, and technological imperatives, which are disregarded by many educational 
providers. 

E-transformation in education 

For the purposes of this study, e-transformation in education is defined as “administrative, pedagogical, and 
technological efforts towards converting traditional learning-including distance learning- environments into e- 
learning environments” . 

According to Bates (2000) and Panda (2003), e-transformation requires visionary leadership, sound policy 
making, planning and operational management, and attention to resource provision, professional development, 
instructional design, student support and research and evaluation (in Latchem, et al. p.616). 

The Economist Intelligence Unit’s (EIU) “The 2003 e-leaming readiness rankings’’ also provide valuable 
suggestions for the dimensions of e-transformation. The EIU suggests that e-leaming readiness is a country’s 
ability to produce, use and expand Internet-based learning and it ranks leading economies across the globe 
according to the following criteria (each of which has its own set of performance indicators: Connectivity (the 
quality and extent of Internet infrastructure). Capability (a country’s ability to deliver and consume e-learning, 
based on literacy rates, and trends in training and education). Content (the quality and pervasiveness of online 
learning materials) and Culture (behaviors, beliefs and institutions that support e-Learning development within 
country). Although this ranking system applies at the country level, these criteria might also be applied with 
some adaptation to judging the e-readiness of institutions. 

According to Latchem (2005) e-transformation is about ensuring that there is visionary and transformational 
leadership, strategic planning and operational planning, and effective management of: 

- access v. cost v. quality 

- change processes 

- curriculum review and development 

- staff development 

- resource and budget allocations 

- technical operations 

- instmctional design and development and delivery systems 

- services to students 

- assessment 

- institutional and program evaluation and quality control 

- recognition and reward systems 

- strategic alliances. 
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Open and distance education and e-transfomation at Anadolu University 

Anadolu University is Turkey’s open university and one of the world’s ‘mega-universities (Daniel, 1996). It 
is a dual mode university with approx. 24,000 on-campus and more than one million off-campus students. The 
University has been providing distance education for 26 years, and has accumulated a great deal of experience 
and expertise in this field. Most of the distance education programs are provided by three Faculties, the Open 
Education Faculty (OEF), the Eaculty of Business Administration (EoBA), and the Eaculty of Economics (FoE). 
OEE assumes the central role in organizing and coordinating different aspects of the distance education 
processes, such as the creation of instructional materials, test production, student assessment, academic 
counseling, TV broadcasting, etc. This Faculty mostly houses two-year associate programs, while FoBA and 
FoE offer Bachelor’s level degree programs. According to the most recent data, OEE has 348,000 distant 
students while EoBA and FoE have 446,000 and 319,000 students, respectively, totaling 1,1 13,000 students. 

Recent figures show that the majority of the distance students are employed (78%), 30% live in villages and 
small towns, 62% are over 24 years old, 45% are married, and 46% are female. More than 900 disabled students 
and more than 700 imprisoned students continue their higher education in the programs (Self Evaluation Report 
of Anadolu University 2008-SER, p.26). 

The University’s distance programs have traditionally been print-based. Students have been, and in many 
cases, still are, expected to study the set texts at their own pace, take self-tests and then sit the scheduled 
centralized exams. Their study is supported by several services including TV broadcasts aired by a state channel 
throughout the country, video and radio programs distributed on various media, face-to-face lecturing for certain 
courses, support through administrative offices, e-learning, and most recently, m-learning or mobile phone 
services. The rationale behind this approach is to provide higher education opportunity to masses in a cost 
effective way (SER, p.26). 

However, online or e-learning has been gaining in popularity for the last ten years. Programs based on e- 
learning include: 

Internet-based Information Management Associate Degree Program as from 2001 
English Language Teaching and Pre-School Teaching Bachelor’s Degree Program-blended as from 
1999 

Anadolu University- Empire State College (SUNNY) Joint e-MBA Program, as from 2003 
Education of Developmentally Disabled Children Non-Thesis Masters Program, as from 2004 
Hospitality Management Non-Thesis Masters Program, as from 2004 
e-Certificate Programs, as from 2006 

The university declares that it aims to provide more student-centered learning environments that lead to 
richer and more effective, collaborative and interactive learning. Parallel to the advancements in the ICTs, many 
new e-learning components and students support services have been introduced (see Table 1). 



Service 



Table 1 : New e-Transformation Related Initiatives of Anadolu University 



Date of ggj.yi(.g Description 
Launch 



Yunusemre 

Project 


2008 


Open Access to 

Learning 

Materials 


2004 


e-Television 


2004 


e-Tutoring 


2004 


e-Audio Books 


2004 


e-Books 


2003 


e-Administration 


2003 


e-Practice 


2003 


e -Exams 


1999 



Open courseware initiative to make available e-components of the 
distance courses to public with no charge. 

Any student enrolled in any on-campus program of the University 
can access to distance learning materials provided in the e- 
Leaming Portal without registering to any distance programs. 
Educational TV programs that are associated with distance 
courses are on the Web. 

Along with the e-practices, this service allows students to ask 
questions to subject experts. 

Prepared especially for visually-impaired students, as well as other 
students who wish to listen to course content. 

Distance course textbooks, converted to PDE file, allowing 
students to read their books on the Web at their convenience). 

Students reach certain services (i.e. registration, issuance of documents, 
grades, etc.) through the Web. 

Practice exercises to help students learn the content better. 

Interactive trial exams to help students prepare themselves for exams. 
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The increasing use of e-books, e-administration, email, telephone, online FAQs and e-tutoring by course 
moderators are reducing the need for student attendance at the OFF centers (Latchem et al, 2006: 227). The e- 
learning portal, where all e-services are presented, has been visited more than 27 million people between May 
2005 and April 2008. Table 2 below indicates the increasing user adoption of selected e-learning components by 
years. 



Table 2: Number of Students Using E-Learning Components by Years 





Content 


2005* 


2006 


2007 


2008** 


Total 


e-Television 


1,780 programs for 178 
courses 


80,475 


180,472 


291,492 


221,499 


773,938 


e-Audio Books 


474 units in 32 audio 
books 


16,789 


88,904 


165,135 


116,717 


387,545 


e-Books 


2,922 units in 232 books 


79,498 


162,625 


319,834 


266,597 


828,554 


e-Practice 


1,029 exercises in 74 
courses 


72,845 


159,523 


229,160 


187,597 


649,125 


e -Exams 


1 1,385 questions for 134 
courses 


118,971 


225,758 


324,393 


260,024 


929,146 



Source: Computer-assisted Learning Unit. * Beginning of May, ** End of April. 



Regarding e-transformation initiatives undertaken at Anadolu University, Latchem et al. (2006:230) observe 
that the University has excellent technology and infrastructure for this work but that its e-transformation process 
still lacks coherent envisioning, strategic planning, quality assurance, faculty development and reflective 
practice. They suggest that senior management team needs to exercise leadership and encourage all faculties and 
staff to align their goals and activities to a new vision, find ways of achieving higher participation in the remote, 
rural and underdeveloped regions, and to take measures to decrease high number of dropout rates. 

Methodology 



The problem 

The University is increasing the number of courses presented via the Web. Some courses are created 
especially for e-learning environments on a learning management platform, although the majority of the online 
learning materials are digital versions of printed texts. Judged on this basis, it is might be argued that “the Open 
Education Faculty has almost realized its aim of e-transforming 90% of the course material” (Source: Self 
Evaluation Report of Anadolu University (SER), prepared in January 2008, for external evaluation by European 
University Association-Institutional Evaluation Programme, p.337). 

The problem here is that the definition of e-transformation in the University is not sufficiently comprehensive 
to realistically evaluate the efforts towards e-transformation. 

The goal of the study 

This paper aims to arrive at a realistic evaluation of the e-transformation efforts taken by the University in 
order to increase the effectiveness of e-transformation process. The study is based, not upon formal reports or 
records but on the university’s personnel’s opinions and perceptions, which is very important since the human 
factor is indispensable part of any change process. 

It is intended that the findings of this study will provide feedback for those in charge of educational 
transformation in Anadolu University to help them plan and implement a more thorough and comprehensive e- 
transformation process. The set of criteria employed for evaluating the e-transformation process in the study may 
be adapted to the needs of other institutions with similar characteristics and work environments. 

Data collection and analysis 

A 5-point Likert type scale questionnaire with 37 items was designed and applied to the academics and 
technical people working in three Faculties (i.e. OEF, FoBA, and FoE), who are stakeholders in e-transformation 
processes, either as administrators, unit directors, faculty members e-learning content developers, instructors or 
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tutors. The reason for including these three Faculties in this study is that they are the central units planning, 
producing, and delivering distance education and the centres where most of the university’s e-transformation 
processes may be said to be occurring. Of the 75 managers and staff whose profiles were considered appropriate 
for this study, 45 persons were contacted and contributed to this study. 

The questionnaire, which was developed from a review of the literature on e-learning, was designed to seek 
information on the following items: Leadership/Management, Connectivity and Infrastructure, Educational 
Climate, Organizational Culture, Capability, Legislative Environment, and User Adoption. 

The demographics of the sample appear in Tables 3 and 4. The participants constituted three main groups: 
managers (faculty administrators and unit directors), lecturers, and technical staff, (see Table 3). The largest 
group of respondents was faculty members using e-learning content (21 staff or 46.7%). 

Table 3: Positions of the Respondents 



Erequency Percent 



Faculty Administrator 


4 


8,9 


Unit director 


6 


13,3 


Faculty member using e-learning content in teaching 


21 


46,7 


E-learning content developer 


14 


31,1 


Total 


45 


100 



Apart from the respondents who had worked in their positions for more than 15 years, average years of work 
for the rest of the groups were shown to be similar. 



Table 4: Years of Work in the Current Position 

Cumulative 

Erequency Percent Percent 



Less than 5 years 


13 


28,9 


28,9 


5-9 years 


12 


26,7 


55,6 


10 - 14 years 


13 


28,9 


84,4 


More than 15 years 


7 


15,6 


100 



A non-parametric Kruskal -Wallis test was carried out in the study (p-value of < 0.05 was considered to 
indicate statistical significance). The reliability of the items was tested by Cronbach Alpha analysis, and found 
highly reliable (.949). One item was removed the data set after the reliability analysis because this was found to 
have negative correlation with the total correlation of the item set. 

Using the Kruskal -Wallis test, statistical discrepancy was sought between demographic indicators and 
responses of the participants and no difference was found for the most of the cases. 

Limitations 

The opinions and perceptions reflected are those of managers and staff in the three Eaculties only (i.e. OEE, 
FoBA, and FoE) even though there are several other Faculties and many more people involved in e-leaming 
activities in Anadolu University. 

For a sample of 45 people, conducting interviews with individuals would seem to be the more appropriate 
means of obtaining useful and accurate data. However, the same question form could be applied to the much 
larger group of people involved in e-learning within Anadolu University in a second phase of the study. For 
comparability of two groups of samples, same tools have to be used for gathering data. 
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Findings 



Table 5 through Table 1 1 presents the responds of participants regarding the extent to which they agreed with 
the statements given under the various specific categories. If any of the statements points to disagreements on the 
scale, this item is given in bold in the Table to indicate problem areas within that category. Indecisiveness was 
treated in the same way, because high levels of indecisiveness implies either that respondents cannot evaluate the 
situation because they do not know anything about it or that in fact no data exists in that area. Figures in bold 
indicate the highest value on the scale. 

Leadership/Management 



Table 5: Respondents’ Opinions and Perceptions on Leadership/Managerial Aspects 





Strongly 

Disagree 


Disagree 


Undecided 


Agree 


Strongly 

Agree 




N* 


% 


N 


% 


N 


% 


N 


% 


N 


% 


The University has vision, mission, and 
plans on e-learning 


2 


4.4 


2 


4.4 


4 


8.9 


23 


51.1 


14 


31.1 


Plans and objectives of the University on 
e-learning are known by the personnel 


4 


8.9 


9 


20.0 


17 


37.8 


10 


22.2 


5 


11.1 


New e-learning practices are started within 
a systematic way 


2 


4.4 


4 


8.9 


12 


26.7 


17 


37.8 


10 


22.2 


The University has specific policies 
regarding the use of ICTs in education 


2 


4.4 


1 


2.2 


13 


28.9 


22 


48.9 


7 


15.6 


The level of coordination among related 
departments is satisfactory 


3 


6.7 


7 


15.6 


23 


51.1 


9 


20.0 


3 


6.7 


The level of cooperation among related 
departments is satisfactory 


3 


6.7 


5 


11.1 


22 


48.9 


11 


24.4 


4 


8.9 


Funding projects and works is not a 
problem 


0 


.0 


2 


4.4 


10 


22.2 


14 


31.1 


19 


42.2 


Human resources are enough in talent 


0 


.0 


9 


20.0 


8 


17.8 


23 


51.1 


5 


11.1 


Human resources are enough in numbers 


2 


4.4 


6 


13.3 


11 


24.4 


19 


42.2 


7 


15.6 


Cost and benefit analysis are used when 
starting new e-learning projects 


1 


2.2 


6 


13.3 


24 


53.3 


7 


15.6 


7 


15.6 


Finding and employing human resources 
are among important problems 


3 


6.7 


14 


31.1 


12 


26.7 


13 


28.9 


3 


6.7 


What the University has achieved so far 


1 


2.2 


7 


15.6 


13 


28.9 


18 


40.0 


6 


13.3 



with regard to e-learning is satisfactory 
♦Number of respondents (frequency) 

While majority of the respondents (Agree and Strongly Agree, 82.2%) strongly believed that the University 
has vision, mission, and plans on e-learning, many of them also are uncertain about whether these plans and 
objectives of the University on e-learning are known by the personnel (Undecided, 37.8%). Most of them (Agree 
and Strongly Agree, 60%) agreed that new e-learning practices are embarked on a systematic way and that the 
University has specific policies in place regarding the use of ICTs in education (Agree and Strongly Agree, 

60%). 

Participants were unclear on whether the level of “coordination” and “cooperation” among related 
departments were satisfactory (Undecided, 51.1% and Undecided 48.9%, respectively). Funding for projects was 
not seen as a problem by the majority of the respondents (Strongly Agree, 42.2%; Agree, 31.1%). Participators 
believed that the human resources were adequate, both in terms of “talent” and “numbers” (Agree, 51.1% and 
42.2%). This finding contradicts with Aydin’s (2005) view that the greatest challenge to extending and 
enhancing e-learning at Anadolu University is a lack of staff knowledge and skills in multimedia and online 
learning. 

53.3% of the participants had no idea whether cost and benefit analysis were used when embarking on new e- 
learning projects. 31.1% disagreed that finding and employing human resources were among important problems 
while 28.9% saw them among important problems. More than half of the respondents (Strongly Agree, 40%; 
Agree, 13.3%) agreed that what the University had achieved so far with regard to e-learning was satisfactory. 
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Connectivity & Infrastructure 



Most of the participants agreed that connection quality and speed of the Internet did not hinder the 
educational processes (48.9%), that the infrastructure was adequate (53.3%), and that working conditions and 
physical areas were adequate (48.9%). However, only 31.1% of the participants believed that students’ access to 
services were taken into consideration while 44.4% were indecisive on this point. 



Table 6: Respondents’ Opinions and Perceptions on Connectivity and Infrastructure 





Strongly 

Disagree 


Disagree 


Undecided 


Agree 


Strongly 

Agree 




N 


% 


N 


% 


N 


% 


N 


% 


N 


% 


Connection quality and speed of 
the Internet do not hinder 


2 


4.4 


3 


6.7 


7 


15.6 


22 


48.9 


11 


24.4 


educational processes. 
Infrastructure (i.e. hardware, 
software) is adequate 


0 


.0 


4 


8.9 


4 


8.9 


24 


53.3 


13 


28.9 


Working conditions and physical 
areas are adequate 


0 


.0 


7 


15.6 


10 


22.2 


22 


48.9 


6 


13.3 


Students’ access opportunities to 
services are taken into 


1 


2.2 


3 


6.7 


20 


44.4 


14 


31.1 


7 


15.6 


consideration 







Educational climate 



Table 7: Respondents’ Opinions and Perceptions on Educational Climate 





Strongly 

Disagree 


Disagree 


Undecided 


Agree 


Strongly 

Agree 




N 


% 


N 


% 


N 


% 


N 


% 


N 


% 


Production of educational material 


0 


.0 


4 


8.9 


14 


31.1 


24 


53.3 


3 


6.7 


is satisfactory in quality 
Production of educational material 
is satisfactory in number 


1 


2.2 


3 


6.7 


10 


22.2 


27 


60.0 


4 


8.9 


Training and support services 
provided for technical and 
academic personnel are 
satisfactory 


2 


4.4 


14 


31.1 


14 


31.1 


14 


31.1 


1 


2.2 


New e-learning initiatives are 
started based on needs analysis 


2 


4.4 


8 


17.8 


18 


40.0 


13 


28.9 


4 


8.9 


Efforts towards increasing 
quality of educational provision 
are satisfactory 


1 


2.2 


13 


28.9 


16 


35.6 


11 


24.4 


4 


8.9 


Faculty who are engaged in e- 
learning are rewarded 


7 


15.6 


8 


17.8 


20 


44.4 


10 


22.2 


0 


.0 


The infrastructure in the University 
allows educational activities are 


1 


2.2 


6 


13.3 


10 


22.2 


22 


48.9 


6 


13.3 


conducted from outside the campus 

Personnel are encouraged for 
making innovations (i.e. novel 


3 


6.7 


13 


28.9 


17 


37.8 


11 


24.4 


1 


2.2 



methods, new programs, new 
approaehes, ete.) 



The majority of the participants agreed that the production of educational material was satisfactory in both 
quality and number (Agree, 53.3% and Agree, 60%, respectively). The question on the training and support 
provided for technical and academic personnel received mixed responses, with about the same percentages of 
participants in the agree, disagree, and undecided categories (31.1%). 40% of the participants were indecisive on 
the issue of whether new e-learning initiatives were based upon needs analysis. 35.6% had no clear idea about 
whether efforts towards increasing quality in educational provision were satisfactory, and only 28.9% did agree 
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that this was the case. 44.4% were indecisive on the point about faculty engaged in e-learning being rewarded for 
this work. 

Almost half of the participants agreed that the infrastructure in the University allowed for educational 
activities are conducted from outside the campus (Agree, 48.9%). On the other hand, 37.8% of the respondents 
were indecisive about there being encouragement for personnel making innovations while 28.9% did not agree 
with this statement. 



Organizational culture 

Table 8: Respondents’ Opinions and Perceptions on Organizational Culture 





Strongly 

Disagree 


Disagree 


Undecided 


Agree 


Strongly 

Agree 




N 


% 


N 


% 


N 


% 


N 


% 


N 


% 


Level of training of aeademie and 
teehnieal personnel with regard to 
e-transformation is adequate 


2 


4.4 


10 


22.2 


18 


40.0 


13 


28.9 


2 


4.4 


Creativity and risk taking are 
eneouraged in the University 


8 


17.8 


15 


33.3 


15 


33.3 


6 


13.3 


1 


2.2 


Cooperative efforts between the 
University and other institutions 
are satisfaetory 


2 


4.4 


9 


20.0 


19 


42.2 


12 


26.7 


3 


6.7 


R&D, evaluation, and quality 
assuranee systems towards e- 
learning are adequate 


4 


8.9 


14 


31.1 


18 


40.0 


8 


17.8 


1 


2.2 


Respondents did not have any clear opinions on the 


issues 


of organizational culture. 40% of the participants 



were undecided on the question of training for e-transformation, 40% on the issue of encouragement for 
creativity and risk taking (33% disagreed on this issue), 42.2% were undecided on the issue of cooperation with 
other institutions, and 40% were undecided on the question of research and development, evaluation, and quality 
assurance in e-learning. 



Capability 



Table 9: Respondents’ Opinions and Perceptions on Capability 





Strongly 

Disagree 


Disagree 


Undecided 


Agree 


Strongly 

Agree 




N 


% 


N 


% 


N 


% 


N 


% 


N 


% 


e-assessment and e-evaluation 
processes used are satisfactory 


3 


6.7 


9 


20.0 


16 


35.6 


15 


33.3 


2 


4.4 


e-examination practices provided for 
students are satisfactory 


0 


.0 


5 


11.1 


13 


28.9 


22 


48.9 


5 


11.1 


Student support services are 
satisfactory 


0 


.0 


2 


4.4 


13 


28.9 


26 


57.8 


4 


8.9 


Institutional determination for 
increasing lifelong learning 
opportunities exists in the University 


1 


2.2 


2 


4.4 


9 


20.0 


21 


46.7 


12 


26.7 



In regard to e-learning capabilities in the University, 35.6% were indecisive about whether e-assessment and 
e-evaluation processes were used satisfactorily, while 33.3% agreed with this statement. 48.9% agreed that e- 
examination practices provided for students were satisfactory. The majority of the respondents (57.8%) agreed 
that student support services were satisfactory while 46.7% believed that there was institutional commitment to 
increasing lifelong learning opportunities. 

Legislative environment 

55.6% of the respondents had no idea whether laws and regulations in Turkey encouraged e-transformation. 
Only 40% felt that regulations regarding e-learning were in place in the University. 
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Table 10: Respondents’ Opinions and Perceptions on Legislative Environment 





Strongly 

Disagree 


Disagree 


Undecided 


Agree 


Strongly 

Agree 




N 


% 


N 


% 


N 


% 


N 


% 


N % 


Laws and regulations in Turkey 
eneourage e-transformation 


5 


11.1 


10 


22.2 


25 


55.6 


5 


11.1 


0 .0 


Regulations addressing issues such as 
copyright, intellectual property 
rights, plagiarism, etc, with regard to 
e-learning are in place in the 
University 


1 


2.2 


6 


13.3 


13 


28.9 


18 


40.0 


7 15.6 


ler adoption 




















Table 11: Respondents’ Opinions and Perceptions on 


User adoption 






Strongly 

Disagree 


Disagree 


Undecided 


Agree 


Strongly 

Agree 




N 


% 


N 


% 


N 


% 


N 


% 


N % 


Level of e-learning opportunities 
provided by the University is 


1 


2.2 


4 


8.9 


12 


26.7 


23 


51.1 


5 11.1 


satisfactory 

Level of m-learning opportunities 
provided by the University is 
satisfactory 


4 


8.9 


6 


13.3 


21 


46.7 


10 


22.2 


4 8.9 



5 1 . 1 % of the participants believed that the level of e-learning opportunities provided by the University was 
satisfactory while 46.7% were indecisive on the issue of whether the level of m-learning opportunities provided 
by the University was satisfactory. 

Discussion and conclusion 

The responses of the participants suggests that many managers and staff are either unfamiliar with the kinds 
of issues involved in e-transformation, or have not yet considered their personal positions on these matters. 

E-transformation involves far more than simply placing existing courses on the Web and providing students 
with e-support services. It needs to be envisioned, planned, operated, and evaluated according to administrative, 
pedagogical, and technological dimensions. Considered in terms of these dimensions, the University’s claim that 
“the Open Education Faculty has almost realized its aim of e-transforming 90% of the course material” should be 
reviewed once more. 

This study has revealed that performance of the University in terms of e-transformation is variable if it is in 
fact judged by the various dimensions suggested by Bates (2000), Panda (2003), Latchem (2005) and the 
Economist’ e-readiness rankings . The University has exhibited satisfactory performance on funding, 
infrastructure, production of e-learning materials, student support services, number and talent of human 
resources. However, in certain other areas, performance of the University appears rather less satisfactory. In 
particular, the organizational culture does not appear to be supporting the University’s personnel towards being 
more innovative, more risk taking, or more reflective in their work. Judging by the research findings where the 
responses of those interviewed reflected relatively high levels of disagreement and indecisiveness, the University 
should reconsider the following in pursuit of a more dynamic and effective e-transformation process: 

Leadership/management 

• Better diffusion and dissemination of the University’s vision, goals and plans on e-leaming 

• A clearer definition of what is meant by, and what is involved in e-transformation and e-leaming 

• More methodical approaches in initiating new e-learning projects 

• Better coordination and cooperation among departments 

• Improved recmitment, training, rewarding and retaining of staff 
Connectivity and infrastructure 

• More consideration of students’ access to services 
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Educational climate 

• Greater concern for quality in curriculum, course content and pedagogy 

• Better training and support for staff 

• More focus on the needs of learners 

• More encouragement for staff engaged in e-learning 

• More encouragement for staff to be innovative and risk takers 
Organizational culture 

• Better preparation of personnel in regard to e-transformation 

• More emphasis on research, evaluation and quality assurance 

• More collaboration with other institutions and possibly other sectors 
Capability 

• Better e-assessment and e-evaluation processes 
Legislative environment 

• More efforts to influence government agencies for more effective laws and regulations on e- 
transformation 

• Clearer policies and guidelines on e-learning and cyber ethics 
User adoption 

• More m-learning opportunities for learners 
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Abstract 

This qualitative case study explored Chinese participants’ online collaboration during group activities. The 
population was drawn from Chinese graduate students enrolled in business programs in U.S. universities. 
Archived online transcripts and interviewing were two major data collection methods in this study. The 
framework of content analysis was based on the Curtis and Lawson (2001) coding schema. The findings 
indicated that Chinese participants exhibited high frequency of contributing, social interactive and seeking 
input behaviors. The behaviors of challenging the input of others and explaining and elaborating one’s own 
position had a relatively low frequency. This study also expanded the findings and explored the perceptions 
and attitudes of Chinese online participants. Their perceptions of gender roles, roles and relations, student- 
instructor interaction, conflict management, motivation, and challenges were greatly influenced by Chinese 
cultural values and beliefs. 



Introduction 

The cultural diversity of learners in U.S. universities has become one of the major issues in 
designing the instructions and implementing the online teaching strategies in a multicultural learning 
context (Rogers, Graham, & Mayes, 2007; Gunawardena & Mclsaac, 2004). 

As an increasing number of international students enroll in online distance learning courses or 
programs, Pincas (2001) argued that U.S. universities might promote online courses too quickly and are not 
ready to deal with problems related to students’ different cultural and linguistic backgrounds in a 
multicultural learning environment. Western models of distance education emphasize individual 
development, autonomy in learners, and active learning (Robinson, 1999, Wang, 2006). Collaborative 
learning and problem-based learning (PEL) are the primary instructional strategies that are employed in the 
distance education in the U.S. In contrast, the Chinese traditional pedagogy, which applies to online 
courses, focuses on teacher-centered and lecture-based knowledge transmission instead of development of 
individual capabilities (Robinson, 1999; Yu, Wang, & Chen, 2005). Since instructional strategies (e.g., 
collaborative learning or problem-based learning) are applied to the online settings in the U.S., Chinese 
international students may struggle due to cross-cultural differences in values, languages, and learning 
styles (Ku & Lohr, 2003; Treuhaft, 2000). 

If U.S. universities want to take the leading role in promoting globalization by means of 
technology, they must understand complex issues such as styles of learning, communication patterns, and 
values and beliefs of culturally diverse learners. Therefore, more research is needed to explore the 
interactive online behavior patterns of international students in order to increase cultural awareness among 
U.S. online instructors and instructional designers (Gunawardena & LaPointe, 2007; Gunawardena, Nolla, 
Lopez-Islas, Ramirez- Angel, & Negchun-Alpizar, 2001; Mason, 2007). 

Theoretical and Conceptual Framework 

The theoretical and conceptual framework for this study was derived from the practice and 
theories of culture and collaborative learning in a problem-based learning (PEL) context. These theories 
and conceptual frameworks served as the theoretical underpinnings for the current study. 

Collaborative learning is based on the social constructivism that emphasizes the importance of 
culture and context. Students construct knowledge based on their understandings about what occurs in 
society. This theory is associated with Vygotsky’s (1962, 1978) concept of social cognition, emphasizing 
that social and cultural interactions have a great influence on one’s cognitive development. 
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Collaborative learning is also consistent with PBL (Barrows, 1986), an instructional method in 
which learners are given ill-structured real-world problems to be solved in order to enhance the content 
knowledge and develop communication, problem-solving, and self-directed skills. Savery and Duffy (1995) 
describe the PBL learning environment as a cognitive apprenticeship environment where scaffolding is 
provided to support learners in constructing cognitive knowledge and developing their metacognitive skills. 
They believe that PBL is consistent with instructional principles derived from constructivism, which 
stresses that knowledge is constructed through experience and social negotiation. 

Matsumoto (1996) perceives culture as “the set of attitudes, values, beliefs, and behaviors shared 
by a group of people, but different for each individual, communicated from one generation to the next” 
(p.l6). Matsumoto believes that “individual differences in culture can be observed among people in the 
degree to which they adopt and engage in the attitudes, values, beliefs, and behaviors that, by consensus, 
constitute their culture” (p.l8). Therefore, culture is an individual, psychological construct as well as a 
social construct. Hofstede (2001) defined culture as “the collective programming of the mind that 
distinguishes the members of one group or category of people from another” (p. 9). Hall (1998) believed 
that culture is communication. “The essence of any culture is primarily a system for creating, sending, 
storing, and processing information. Communication underlies everything” (p. 53). 

Hofstede (1980, 1984) divides culture into four dimensions: individualism-collectivism, 
uncertainty avoidance, power distance, and masculinity-femininity. In addition. Hall (1966, 1976) develops 
the theory of intercultural communication and introduces high-context and low-context cultural factors 
with an emphasis on a strong linkage between culture and communication. The theory distinguishes 
communication patterns among different cultures based on the information that is implied versus stated 
directly in a communicative message. 

Purpose of the Study 

The purpose of this study is to explore (1) the online collaborative behaviors among a group of 24 
graduate-level Chinese international business students when engaging in solving an ill-structured problem 
and (2) how cultural values and beliefs influence their collaborative online behaviors. 

This study seeks to answer the following research questions: 

• Will Chinese international students exhibit collaborative online behavior(s) when working as 
a group on an ill-structured problem? If so, what type(s) of collaborative online behavior do 
Chinese students engage in? 

• How cultural values and beliefs influence their online collaboration? 

Research Method 

This study involved six groups of Chinese graduate business students with four members in each 
group who engaged in the same online simulated activity over two weeks. A qualitative research design 
that integrated exploratory case study was used for this study. 

Participants 

The participants were 24 graduate students enrolled in business programs from 12 U.S. 
universities within four states. Twenty of them were from universities in the Midwest (N=20). Four of them 
were from universities on the East coast (N=4). The participants reported that they had completed their 
undergraduate programs in Chinese colleges or universities and were currently studying in graduate 
business programs in U.S. universities with concentrations on general management, economics, accounting, 
finance, marketing and, etc. The participants were selected purposely to represent a diverse range of 
characteristics such as institutions, gender, and level of language proficiency. The participants were 
assigned to six online learning groups with group size of four members during the PBL activities. The four 
members in each group were geographically dispersed at different universities. In addition, this study 
included 12 out of 24 participants for follow-up interviews. 



203 




Data Collection 

For the purpose of this study, data were collected mainly from two primary data sources: archived 
online transcripts and interviews. Data collection took place from June to December 2007. During each 
group’s activity, the researcher observed their online interactions by reading through the ongoing 
asynchronous discussion forums. Follow-up Interviews allowed the researcher to pursue in-depth 
understandings about the interviewees’ perceptions and attitudes about their online collaborative 
experience. Thus, interviews served the purpose of clarification and verification. The interview data also 
triangulated the online transcript data to provide meaningful interpretations as to why participants 
interacted collaboratively in a certain way. 

Data Analysis Process 

This study mirrored the Curtis and Lawson (2001) coding scheme of analyzing collaborative 
learning behaviors to explore Chinese students’ online collaborative behaviors in an asynchronous online 
environment. This schema provides five behavioral categories and 15 behaviors to code the online 
transcripts, which are derived from the nine components of collaborative behaviors described by Johnson 
and Johnson (1996) who identified the theoretical perspectives of collaborative learning: planning, 
contributing, seeking input, reflection/monitoring, and social interaction. 



Table: The Curtis and Lawson (2001) Coding Schema 



Behavioral Categories 


Codes 


Planning 


Group Skills (GS) 


Organizing Work (OW) 


Initiating Activities (lA) 


Contributing 


Providing Help (HeG) 


Giving Feedback (FBG) 


Exchanging Resources and Information 
(RI) 


Sharing Knowledge (SK) 


Challenging Others (Ch) 


Explaining or Elaborating (Ex) 


Seeking Input 


Seeking Help (HeS) 


Seeking Eeedback (EBS) 


Advocating Effort (Ef) 


Reflection / 
Monitoring 


Monitoring Group Effort (ME) 


Reflecting on Medium (RM) 


Social Interaction 


Social Interaction (SI) 



Three researchers were involved in the online transcripts’ coding process. Six groups’ archived 
online transcripts containing a total of 347 posting and messages were analyzed by two of the coders while 
a third coder arbitrated the situations in which previous two coders could not reach the consensus. 

The conceptual framework of interview data analysis is influenced by the Miles and Huberman 
model (1994) and the constant comparative method (Glaser & Straus, 1967; Merriam, 1998). To ensure 
validity and reliability, this study calculated the inter-rater reliability and utilized different methods, 
including triangulation, member checks, peer debriefing, and the translation-back translation strategy. 

Results 

The research findings revealed that six groups displayed the highest frequency of the contributing 
behavior, followed by the behaviors of social interaction and seeking input. The behaviors of challenging 
the input of others (CH) and explaining and elaborating one’s own position had a relatively low frequency. 
In terms of the planning behavior, those who displayed more evidence of the planning behavior emerged to 
become group discussion leaders. The behaviors of reflection/monitoring exhibited relatively low 
frequency, focusing mainly on the discussions around the communication tools that participants chose to 
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use for completion of the group projects. Some interview data supported the findings from the online 
transcripts. 

Interviews further investigated contributing factors (e.g., socio-cultural, linguistic, and personality) 
that had an impact on Chinese participants’ online behavior patterns. Some of the themes emerging from 
these topics: Chinese participants 

• believed that collaborative learning was an effective way to learn, but they need to adapt these 
learning styles; 

• preferred face-to-face settings vs. online settings for learning; 

• acknowledged that the group leader’s role was critical but they were reluctant to assume this role; 

• tried to preserve harmony during conflict resolution; 

• preferred using both asynchronous and synchronous ways to communicate within groups; 

• identified six types of challenges during online collaboration (e.g. language and cultural influence, 
lack of motivation, lack of leadership skills; lack of high-level social interaction; different levels 
of knowledge; and lack of time management skills); 

• expected their online instructors to have culture-sensitive knowledge; 

• advised future Chinese online learners to have proactive attitudes. 

Some of Chinese participants’ thoughts about how they behaved during the group process and 
development were greatly influenced by the Chinese cultural beliefs and values. For example, the Chinese 
culture influenced their perceptions of crucial elements in collaborative learning, the online communication 
patterns, the roles of instructors and group leaders, conflict management, motivation, social interaction, and 
many other issues in group dynamics. 



Discussion 

The findings reinforce that language and culture are the two main factors that influence Chinese 
participants’ online collaborative behaviors. Specifically, participants’ limited language proficiency affects 
their confidence and effective communication during their online activities. 

The high frequency of contributing behaviors among Chinese participants indicated that Chinese 
participants perceived that the individual’s contributing behavior was the most basic requirement for the 
group members. However, the behaviors of seeking help (FBS), challenging the input of others (CH), and 
explaining and elaborating one’s own position (Ex) had relatively a low frequency. Chinese participants 
hesitated to challenge the ideas of others and express different opinions in order to save others’ face and 
create a harmonious learning atmosphere. They were reluctant to engage in higher-order thinking in the 
online environment due to the lack of high-level social interactions in the group. The finding further 
supports previous research conducted by Tam and Bond (2002) that Chinese culture puts more value on 
friendship and family so that close interpersonal relationship can only be viewed within a longer time 
frame. 



Findings also revealed that Chinese participants felt more favorable when learning in the face-to- 
face setting. Several reasons might cause this phenomenon. First, the lack of familiarity with the online 
learning environment might hinder Chinese participants for their online engagement (Mason, 2007). 
Second, Chinese participants were accustomed to learning passively and did not realize the changing role 
between instructors and students (Lee, 2004). Third, Chinese students might not appreciate the 
constructivist learning philosophy that emphasizes the learner-centered approach since the teacher-centered 
approach is widely accepted and applied in the Chinese educational system (He, 2001). Thus, Chinese 
participants might not be ready psychologically to take online courses. 

The results of the findings also supported Hofstede’s (1980, 1984) four dimensions of culture and 
Hall’s (1966, 1976) contextualization. 
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Collectivism vs. Individualism 

Chinese participants were found to be more collectivist-oriented during their online group 
processes and development. This reflects that Chinese people are likely to be interdependent and be 
connected with each other socially and psychologically (Hsu, 1953). 

Power Distance 

Hofstede (2001) indicates that the Chinese culture displays much larger power distance than the 
western culture. The findings revealed that Chinese participants hoped their group leaders could assume the 
role of being an instructor to take the responsibility of guiding the group and help group members with 
their cultural adaptation in the online environment. In addition, Chinese participants hesitated to take the 
leading role if group members were older than him/her. The findings are consistent with the conclusion 
drawn from Hu & Grove (1991), “Throughout their history, the Chinese have shown respect for age, 
seniority, rank, maleness, and family backgrounds” (p. 6). 

Avoidance Tolerance 

For this study, Chinese participants preferred a structured collaborative online learning 
environment under teachers’ guidance. When the instructor is absent in an online environment, they like to 
closely follow the directions from their leader. In addition, Chinese online participants preferred 
periodically review what they learned or their progress to ensure that they were on the correct path. The 
findings from this study coincide with those from Hofstede (2001) who studied the cultures with high- vs. 
low-uncertainty avoidance. 

Masculinity-femininity 

Growing up in the feminine culture, Chinese online participants liked to collaboratively learn to 
help their group members work through disagreements. They searched for the empathetic feelings when 
dealing with conflicts. Making decisions to participate in the activity resulted in their intrinsic interest in 
the subjects. Chinese female participants were respected and they were treated equally with the male 
students. Chinese participants’ beliefs and behaviors are in agreement with what Hofstede (2001) describes 
about the masculinity vs. femininity culture. 

High- Vi. Low-context Communication 

According to Hall (1976, 1998), Chinese students who fall in the low-context culture rely heavily 
on nonverbal communication and tend to use silence and ambiguous language to deliver meaning of their 
messages. For this study, Chinese online participants reported that they tended to post their messages after 
carefully thinking and pondering. This reflected the Chinese saying, “thinking thrice before acting.” For 
Chinese culture, nonverbal communication, including facial expression and tone of voice are silent. One of 
the participants interpreted the meaning of her silent classroom behavior as the process of listening, 
reflecting, gathering thoughts, and thinking thoroughly about how to respond to others. Chinese students’ 
silent behaviors are rooted in Asian culture that perceives silence as a way to communicate indirectly, to 
show respect and conformity, and to take time to think thoroughly to integrate diverse perspectives into a 
workable solution without the risk of losing face (Brislin, 2000; Liu, 2001). 

When managing and resolving conflict, Chinese online participants tried to create and preserve a 
harmonious relationship with others. Their conflict management style was in line with the findings from 
Chung (1996) and Ma (1992), who described that people in high-context cultures tend to be more 
nonconfrontational and indirect. When handling conflicts, they do not need to accept different views. But 
they should always respect others and try not to fight with them and “maintain interpersonal harmony 
within their group” (Hwang, 1987, p.952). 

Recommendations for Fntnre Practice 

For U.S. online instructors, they should understand Chinese students’ backgrounds before teaching 
online courses, encourage reciprocal two-way communication, provide an active and helpful learning 
environment to motivate students to learn, and develop and implement appropriate instructional strategies 
for helping Chinese students during their adaptation. 
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For the instructional designers, they should identify needs and perform learner analysis, and 
design an authentic learning environment based on the constructivist theory. 

For the Chinese online learners, they should try to understand American cultures, develop their 
reflective thinking, have a proactive attitude, and spend more time planning for group activities. 

Furthermore, future researchers should focus on different areas: instructor-student interaction, a 
comparative study using a blended sample of Chinese and U.S. students, and perspectives of Chinese 
instructional designers. In addition, development of new schema or models for analyzing online 
collaborative behaviors is necessary to lay the foundation to advance study in the cross-cultural 
communication field. 
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Abstract 

This study incorporated the Zone of Proximal Development based scaffolding into the Adaptive Control of Thoughts 
theory and developed a unified conceptual framework for creating a new feedback strategy called intelligent 
elaborative feedback (lEF). The effectiveness of the lEF intervention on learning performance was examined in a 
computer assisted instructional program for Chinese vocabulary acquisition. Forty-three elementary school students 
enrolled in an after-school program introducing Chinese culture and language participated in the study. A mixed- 
method research design was used to incorporate a randomized pretest-posttest control group design and a combined 
method of think-aloud observation and retrospection focus groups. Results from the repeated-measure ANOVA on 
both experimental group (with IFF condition) and control group (with traditional corrective feedback condition) 
indicated no significant difference in recall and retention performance change between the two groups. Follow-up 
observation and focus groups data supported the results and yielded additional findings on design and 
implementation guidelines and principles of the IFF strategy. 

Introduction 

With the increasing use of computer programs in education, researchers have been investigating the roles of 
scaffolding in teaching and learning (Fajoie, 2005; Quintana et ak, 2004; Sherin, Reiser, & Edelson, 2004; Wells, 
1999; Wood & Wood, 1996). Among them. Wells discussed scaffolding in terms of its connections to the Zone of 
Proximal Development (ZPD, Vygotsky, 1978) and computer-assisted knowledge acquisition. According to Wells, 
the role of artifacts, including computer programs that mediate knowing, was identified as one of the essential 
features of educational scaffolding. 

Wood and Wood (1996) recognized Anderson’s Adaptive Control of Thought (ACT) theory and 
reexamined the metaphor of ZPD-based scaffolding in the context of computer/intelligent tutoring systems. They 
suggested Vygotskian researchers need vision of integrating models of the learning process such as the ACT models 
of cognition when designing and creating scaffolds in computer-assisted tutoring systems. 

As one of the instructional events defined by Gagne (1985), providing feedback is a necessary condition for 
learning and should be considered as one of the basic elements in designing instruction and selecting appropriated 
media (Gagne, Briggs, & Wager, 1992). This study, therefore, developed a conceptual framework based on ACT’s 
elaboration model and ZPD-based scaffolding and created a feedback intervention - intelligent elaborative feedback 
in a computer-assisted language learning program. The effectiveness of such feedback, design and development are 
then investigated. 



Review of the Relevant Literature 



Feedback as a Scaffolding Strategy 

Among the studies investigating the roles of feedback in affecting learning performance, some of them 
suggested or implied the feedback afforded by the computer-based learning systems could serve as a scaffolding 
strategy in promoting learning (Clarebout, Elen, Johnson, & Shaw, 2002; Corbett & Anderson, 2001; Kim & Baylor, 
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2006; Koedinger & Corbett, 2006; Mathan & Koedinger, 2005; Moreno, 2004; Moreno & Mayer, 2005; Moreno & 
Valdez, 2005). Among these studies, Kim and Baylor (2006) proposed a theoretical foundation for creating “more 
functional and more intelligent” (p. 573) pedagogical agent as learning companions (PALs) in the computer-assisted 
learning environments. In their social-cognitive framework, feedback was identified as one of the key constituents of 
PALs and was suggested to be created as a ZPD scaffolding function of PALs to promote learning performance and 
learners’ self-efficacy. Guidelines for designing effective feedback were suggested by Kim and Baylor, as well as 
the directions for future research. 

Integrating scaffolding strategies and creating such activities through feedback to promote deep 
understanding have also been recognized and are under investigation in the cognitive tutor systems (Corbett & 
Anderson, 2001; Koedigner & Corbett, 2006; Mathan & Koedinger, 2005). Mathan and Koedinger recast the 
instructional design debate on immediate feedback and delayed feedback by offering an intelligent novice (IN) 
model of feedback to scaffold students’ learning. This IN model was implemented in the context of Spreadsheet 
Cognitive Tutors based on Anderson’s (1993) ACT theory. Different feedback strategies were used in the IN model, 
where the observable consequences of students’ mistakes served as a form of feedback in error detection exercises to 
allow reasonable errors by the novice learners, and immediate corrective feedback was used in error correction 
exercises to prevent floundering. Pretest-posttest experiments were conducted between the IN model group and the 
traditional ACT expert model group for which only immediate corrective feedback was provided. Results showed 
significantly better performance in terms of problem solving, conceptual understanding, transfer and retention 
among the students receiving IN feedback. Mathan and Koedinger concluded that the IN feedback is an effective 
means to promote deeper metacognitive skills and further considered the IN model as a theoretical advance to the 
ACT theory of cognition by adding the scaffolding functions to promote student performance. 

Both Kim and Baylor’s (2006) social-cognitive view of feedback and Mathan and Koedinger’ (2005) 
intelligent novice model of feedback suggested creating feedback as a scaffolding strategy, which provided strong 
support for the feedback intervention in the present study. In addition, the holistic guide for designing effective 
PALs (Kim & Baylor) and the ACT theory based study (Mathan & Koedinger) had significant theoretical and 
practical values in informing this study. 

Feedback Modality and Learning Outcomes 

When defining feedback in computer-based instruction. Wager and Wager (1985) referred to any message 
or display provided by the computer in response to the learner’s actions. From this viewpoint, feedback could have 
more modalities, such as precision, timeliness, guidance and motivational messages, rather than the traditional 
corrective messages indicating right or wrong answers (Hoska, 1993; Sales, 1993). Four different modalities of 
feedback support afforded by the computer assisted instructional systems were identified by Clarebout, Elen, 

Johnson and Shaw (2002), which include executing, showing, explaining, and questioning. Wager and Wager’s 
(1985) definition of computer based feedback implies that any computer assisted instructional system would utilize 
one or more of these modalities as a form of feedback support. The synthesis of the multimodal form of feedback in 
various studies will follow this assumption and identify the relevance with the present study. 

Investigation on the effects of different modalities of feedback on learning performance and outcomes have 
been performed and represents an ongoing trend in educational research through many recent educational 
technology and educational psychology studies (Anderson, Corbett, Koedinger, & Pelletier, 1995; Baylor & Chang, 
2002; Chuang & O'Neil, 2006; Clariana & Koul, 2006; Koedinger & Corbett, 2006; Ge, Chen, & Davis, 2005; 
Gilbert, Wilson, & Gupta, 2005; Grant & Courtoreille, 2007; Heift, 2006; Van Eck, 2006; Yamashiro & Dwyer, 
2006). Among these, the nonverbal feedback is often considered as a form of affective feedback to meet learners’ 
emotional needs and promote learning motivation and attitudes (Baylor & Chang, 2002; Gilbert, Wilson, & Gupta, 
2005; Moreno, 2004; Moreno & Elowerday, 2006). Moreover, research studies also showed the great needs for 
incorporating cognitive or performance feedback modalities into the affective domain (Clariana and Koul, 2006; 
Gilbert, Wilson, & Gupta, 2005; Moreno & Elowerday, 2006). 

The considerations of cognitive and performance feedback have been examined by many studies (Anderson, 
Corbett, Koedinger, & Pelletier, 1995; Chuang & O'Neil, 2006; Ge, Chen, & Davis, 2005; Grant & Courtoreille, 
2007; Heift, 2006; Van Eck, 2006; Yamashiro & Dwyer, 2006). A study by Ge, Chen and Davis (2005) investigated 
the effects of question prompts in scaffolding novice instructional designers solving ill-structured, instructional 
design problems in a Web-based learning environment. The effects of different prompting conditions (question- 
elaboration vs. question-guidance) were examined, with the considerations on learners’ prior knowledge and 
experience. A comparative multiple-case study design was used in this study. The qualitative findings indicated the 
advantages of question prompts in scaffolding ill-structured problem solving. Also, the specific cognitive and 
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metacognitive functions, as well as limitations, of question prompts in different conditions were discussed and 
implied from this study. The questioning modality with explaining and showing in two different conditions in this 
study shows the significance of utilizing multiple modalities of feedback support in both cognitive and affective 
domain. 

Another study by Heift (2006) examined the effects of three independent factors on student usage of 
context-sensitive help: feedback, exercise type, and language proficiency in the context of computer assisted 
language learning (CALL). Two types of corrective feedback were used in this study: meta-linguistic feedback 
(highlighting error with a detailed explanation) and repetition feedback (error highlighted with only a hint of error 
category). Showing and explaining functions were supported in meta-linguistic feedback, while showing and 
questioning in repetition feedback. Results of this study show that, students are more likely to access additional help 
options when less system feedback is given in meta-linguistic mode. The study suggests that multiple modalities of 
feedback support might interfere with the learners’ actual use of the support tools offered by the computer-based 
instructional system. 

By viewing feedback as a scaffolding strategy and by exploring the effects of different feedback modalities 
on learning outcomes helped to inform the present study and the design of the feedback intervention, as well as to 
provide literature support for determining the assumption and hypotheses of the study. 

Intelligent ZPD-Elaboration; A Unified Theoretical Framework 

The elaboration model in the ACT theory states “to-be-remembered material and any elaborations are 
encoded into a propositional network” (Bradshaw & Anderson, 1982, p. 166). Related conditions in such networks 
produce the best recall and retention performance. The stronger two knowledge pieces (facts) relate to each other, 
the better recall and retention. However, the authors suggested a novice learner may take longer to effectively use 
elaboration from the networks due to redundancy. The need to “prescribe scaffolding activities to help students 
develop a deep understanding of domain knowledge” (Koedinger & Corbett, 2006, pp. 62-63) in ACT-R was 
identified. 

The term scaffolding was introduced by Wood, Bruner and Ross (1976) after parallels with ZPD were 
recognized (Rogoff & Wertsch, 1984; Wood & Wood, 1996). They offered a list of scaffolding functions, among 
which reduction in degree of freedom offers limited yet focused assistance. Fading of the scaffold was introduced by 
Collins, Brown and Newman (1989), which gradually reduces the support as learners attain more knowledge. It 
allows learners to gradually take control of their learning, which reflects Vygotsky’s notion of internalization in 
ZPD. 

This study views reducing freedom and fading scaffolding as the connection points of incorporating ZPD- 
based scaffolding into the ACT theory. First, according to ACT, two knowledge pieces with weaker relationships in 
the network require more learning assistance that indicates the learner falls into the Zone. From this standpoint, 
concepts of ZPD-paths and non-ZPD-paths are defined to distinguish learning conditions. ZPD-paths are the 
elaborative paths between two knowledge pieces in the knowledge network with weaker relationships where 
learners need assistance (scaffolding) in recall and retention. Likewise, non-ZPD-paths are those with stronger 
relationships between knowledge pieces where learners need less or no assistance in recall and retention. 

Second, non-ZPD-paths are eliminated from the inferential knowledge network to reduce the freedom of 
learning. This process helps retrieve information better and faster, which leads to better recall and retention 
performance. In addition, fading scaffolding is naturally integrated and reflected in the process of non-ZPD-path 
reduction. Figure 1 shows the unified theoretical framework. 

Purpose of the Study 

By incorporating the concept of ZPD into the cognitive model of elaborative knowledge processing, this 
study created the intelligent elaborative feedback (IFF) and investigated the effectiveness of IFF in a computer 
assisted instructional program for Chinese vocabulary acquisition among young learners. Intelligent elaborative 
feedback is defined as a form of feedback that provides an appropriate amount of learning support covering content 
elaboration and contextual assistance based on the learner’s status in each step. Therefore, this study proposed and 
answered the following research questions: 

1 . Do the learners who receive intelligent elaborative feedback (IFF) in computer assisted vocabulary 
learning show greater gains than those who receive traditional corrective feedback (TCR) in such 
learning environments? 
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2. What factors are critical in designing and implementing the lEF strategy to enhance the effectiveness 
of this intervention? 

The null hypothesis is that there is no significant difference in the learning performance over time between 
the TCR condition and the lEF condition in the computer assisted vocabulary learning. In addition, the second 
research question served as a guiding question in examining the design attributes and their effect on the intervention. 

Research Method 

This study used a two-phase mixed method. In Phase I, a randomized pretest-posttest control group design 
was utilized; the participants were randomly assigned to two groups: the experimental group with intelligent 
elaborative feedback (lEF) and the control group with traditional corrective feedback (TCR). The independent 
variable was the feedback condition (lEF vs. TCR). The dependent variables were the scores on the vocabulary 
achievement tests, including one pretest (right before the treatment), one posttest (immediately after the treatment), 
and one follow-up test (two weeks after the treatment). In addition, a demographic survey was conducted to collect 
data about each participant’s general information, family background and prior knowledge in the subject domain. 

A follow-up phase (Phase II) was conducted utilizing a combined think-aloud observation and retrospection 
focus group for more in-depth consideration of what elements of lEF are essential in causing the learning 
performance change. The participants in the lEF group were observed and later asked about reactions and their 
thinking and learning processes with a focus on feedback intervention. 

Figure 1. Intelligent ZPD-Elaboration: The Unified Theoretical Framework 
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Participants 

The participants were 43 elementary school students enrolled in an after-school program named “Year of 
the Rat (YTR),” which was offered by the first author to introduce Chinese language and culture at two participating 
school sites in Midwest urban and suburban areas. The age range of the participants was from 6 to 13. Among them, 
16 were male and 27 were female. In Phase I, participants were randomly assigned to two groups: 22 in the 
experimental group with lEF and 21 in the control group with TCR. In Phase II, 3 experimental group participants in 
each school site formed the focus group on the voluntary and first come first serve basis. 

Computer Programs 

The computer program for Chinese vocabulary learning was named “Learn with Michael (LWM).” The 
design of LWM was based on the state learning standards for foreign language beginners, with a focus on the 
vocabulary recognition of basic daily words, such as family members, colors and numbers (ILS 28C.A in ISBL, 
n.d.). Two versions of LWM were developed by the first author using Authorware 7.01. One is the ILL version for 
the experimental group. The other is the TCR version for the control group. Two versions have identical learner- 
computer interaction interfaces and instructional sequences. The only difference was the exercise section where the 
different protocols of exercise question generation and feedback provision were integrated. Figure 2 shows the 
screen shots of two versions of exercise page. 
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Figure 2. Screen shots of exercise page in TCR version and lEF version. 
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Pretest, posttest and follow-up test were embedded in LWM program in the form of vocabulary matching 
games. Three tests had identical items but with different item order. The test protocol was also based on the ILS 
28C.A. The screen shot of the pretest game is shown in Figure 3. 

Figure 3. Screen shot of the pretest; a vocabulary matching game 




Procedures 

The Phase I activities took place in the four-week long YTR after-school program. One session was offered 
each week. The participants took the demographic survey in the first week. In the second week, the participants used 
the LWM computer program, which included a ten-minute pretest game at the beginning, the instruction with 
treatment (either IFF or TCR), and a ten-minute posttest game at the end. In the fourth week of YTR, the 
participants took the follow-up test by playing the vocabulary matching game again. The scores, answers, and time 
spent on the tests were recorded by the LWM program in a text file. In addition, the experiment sessions in the 
second week were video recorded for later observation. 

Two weeks after the YTR program, two focus groups were conducted with one in each school site. Three 
experimental group participants at each school site were grouped together to talk about their reactions and feelings 
about the LWM program and the overall learning process in LWM. 
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Demographic Survey 



The demographic survey asked 13 questions about each participant’s general information (age, gender, 
grade level), prior knowledge and experience in China and Chinese language, and family background in China and 
Chinese language. 

Chinese Vocabulary Achievement Tests 

Three Chinese vocabulary achievement tests (pretest, posttest, follow-up test) were embedded in the LWM 
program. They had identical items, except for the order of the items. The test items took the advantage of the 
performance assessment items suggested in ILS 28C.A, which suggests including 8 items in target language (in this 
case, in Chinese) for young learners and asking the learners to match each item to the English equivalent within 10 
minutes. In each test, the correct answer for each word scored 1 point. The total score for each test, therefore, was 8 
points. Figure 3 shows an actual screen of the tests. 

Focus Group Protocol 

The focus groups were semi-structured. Starter questions were prepared with a focus on the LWM program 
and the feedback intervention. The process took around 40 minutes. The LWM program was running during the 
focus groups to assist the participants in recalling their think-aloud actions in the experiment and verbalizing their 
thoughts during the retrospection process. The responses and the structure of the communications during the focus 
group process remained open to encourage young participants to communicate more about their thoughts and 
reactions to the program. The focus groups were voice recorded. 

Video Observation Protocol 

Actions of the participants in the experimental group were video recorded during the experiment. The video 
records were observed immediately after each session. The observation protocol was open-ended with a focus on the 
participants’ actions on each page/screen and time spent in each activity in the LWM program. 

Results 

For the Phase 1 investigation, a one-way ANOVA was performed to assess the equivalence of the 
experimental group and the control group prior to the treatment. A repeated-measures ANOVA was performed to 
examine the different learning performance changes between two groups over time. 

There are three basic assumptions required for ANOVA tests (Gravetter & Wallnau, 2004). Assumption 1; 
sample values are independent. The assumption was satisfied because participants were from two elementary 
schools and randomly assigned to two groups. Assumption 2: The scores in the population are normally distributed. 
This assumption was met when considering the learning achievement growth from pretest to posttest and the growth 
from posttest to the follow-up test. Assumption 3: the population distributions for each treatment have homogeneity 
of variances. This assumption was met by performing Levene’s test of homogeneity of variances (Levene’s statistic 
W = 0.847), which was not significant at the .05 alpha level {p = 0.363). In addition, Mauchly’s sphericity test for 
the repeated measures variable time did not significantly violate the sphericity assumption {W = 0.859, X^(2) = 4.999, 

p=0.082). 

The results of the one-way ANOVA on the pretest score indicated that there was no significant difference 
between the experimental group and the control group before the treatment (L(l,41) = 0.004, p = 0.974, t]^<0.001). 

The results from the repeated-measures ANOVA tests failed to reject the null hypothesis; that is, there is 
no significant interaction between time and feedback conditions at the .05 alpha level (L(2,68) = 0.697, p = 0.501. 

See Figure 4). The effect size was small (t]^=0.009). In addition, the tests of between-subject effects at the .05 alpha 
level indicate there is no significant difference in the vocabulary achievement change between the ILL condition and 
the TCR condition (L(l, 34) = 0.186,p = 0.669, tj^= 0.005). 
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Figure 4. Mean plot of test scores and interaction between time and feedback conditions 
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In the follow-up phase, focus group voice records were transcribed and coded using the open coding 
technique to discover the emergent concepts and themes (Strauss & Corbin, 1998). Four themes were identified 
during the coding and categorizing process: (1) attitude and interests in learning Chinese language and culture; (2) 
perception of the LWM program; (3) perception of the feedback component; and (4) reactions to the tests 
(vocabulary matching game). The third theme about the feedback component had two sub-themes: usage of the 
feedback and attributes of the feedback. 

First, all of the six participants in the focus groups showed great interest and positive attitude in learning 
Chinese language and culture. In addition, all of them confirmed that they had limited exposure to the Chinese 
language and no prior knowledge of the particular vocabulary set used in the experiment. 

Second, all of the participants favored the LWM program in general and each individual had favorite 
component in the program. When considering the tests (the vocabulary matching games) as part of the program, the 
games were the most favorable component in LWM. And the story about a Chinese traditional dessert, which the 
exercises were based upon, won second place as the favorite. 

Third, five out of the six participants thought the feedback helped their learning process, but not for recall 
and retention (in posttest game and follow-up test game). Regarding the usage of the feedback component, five 
participants confirmed they read the corrective part of the feedback, which indicated whether their answer was 
wrong or right and provided the correct answer if needed. Only one participant read the elaborative part of the 
feedback. The other five admitted they did not read the explanation of each answer. And the color attribute of the 
feedback (green background color for correct answer and its elaboration; pink for wrong answer and its elaboration, 
shown in Figure 2 IFF version) was the reason for the one who actually used the elaboration. 

Fourth, there were mixed reactions to the tests (the vocabulary matching games). While the engaging nature 
of the games gained the interests and attention of all the six participants, two out of the six considered the game 
difficult and two used “random guess” as the strategy in playing the game. Two participants also noticed the 
different order of the test/game items. They claimed they would do better in the latter two games if the order stayed 
the same. In addition, two participants were aware of the time granted for the tests/games. One of them claimed that 
he felt the time would not be enough for him to finish the game so he rushed by using the “random guess” strategy. 

Observations on the video records supported the findings from the focus groups in ways that most of the 
participants in the experimental group did not actually read the elaborative feedback part. Rushing through the 
LWM program by spending very short amount of time in each section was a common phenomenon during the 
experiment. In addition, the peer pressure became a major distracting factor when one saw the others finishing the 
program and started to rush through the rest of the program, which usually covered the exercises with feedback 
component and the posttest game. 
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Discussion and Conclusion 



Providing adaptive and effective feedback to the learners is one of the essential events in any instructional 
situation (Gagne, 1985). From a feedback-as-ZPD-scaffolds perspective and with the intention to effectively 
promote learning performance of recall and retention, this study proposed a theoretical framework, intelligent ZPD- 
elaboration, and created a new feedback strategy, intelligent elaborative feedback (IFF). The effectiveness of IFF 
was then investigated in a computer-assisted Chinese vocabulary learning program. 

The quantitative data obtained from this study indicated there was no significant difference between the 
IFF condition and the traditional corrective feedback condition over time. This result based on a preliminary finding 
about the equivalence between the experimental group and the control group. Although the two groups were similar 
in their demographics and prior knowledge, further analyses will be performed to determine whether the age, school 
or gender played a role in the learning process. Qualitative analyses in this study discovered evidences of what 
caused the above non-significant results, such as the shorter duration of focused attention among young learners, the 
connections between personal interests and the learning process, and the different styles of learning among children, 
which were consistent with the studies on young learners’ characteristics and learning styles in child development 
and educational psychology literature (Cowan, Nugent, Flliott, Ponomarev, & Saults, 1999; Gardner, 1999; Languis, 
1981). Further analysis will also look into the some of the design attributes of computer-assisted instructional 
systems, such as the multimedia attributes and the cognitive load considerations (Mayer & Moreno, 2003; Paas, 
Renkl, & Sweller, 2003) to assist deeper understanding of the result and to provide more findings and implications. 

In addition, future study may need to taking the above considerations into account when designing and 
implementing the IFF strategy in computer-assisted instruction. In addition, investigating such feedback strategy in 
different settings and contexts with older learners may have different results. 

In closing, this study based its research stance on the latest definition of the educational technology field 
and focused on improving individual learner performance regarding obtaining transferable knowledge through 
creating, using and managing a technology-enhanced educational intervention. Motivated by the increasing demands 
for Chinese language learning, the researchers in this study have a vision that parallels the dynamic culture of the 
educational technology field as it makes contributions and satisfy the ongoing demands of the globalization of the 
educational arena. 
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Abstract 

Online learning environments are no longer restricted to two-dimensional formats with linear text and threaded 
discussions. Novel technologies— including avatars, podcasts, and multimedia experiential learning activities— give 
educators an opportunity to exercise their passion for the art of teaching while engaging students and meeting 
curriculum guidelines. This paper illustrates the use of these technologies in an online teacher credential course and 
examines the impact on student learning 
Keywords: teacher education; 

Online learning environments are no longer restricted to two-dimensional formats with linear text and 
threaded discussions. Novel technologies give us an opportunity to embrace the art of teaching while engaging 
students and meeting curriculum guidelines. This paper examines the use of an avatar, podcasts, a virtual tour and 
multimedia experiential activities to create an effective online course in a teacher credential program. It also 
investigates the impact of these novel technologies on student learning. 

Educators who are passionate about the art of teaching understand our K-12 teachers and credential 
candidates are educating a generation born in the computer age. Technology is second nature to the pupils in their 
classes. By the time these children started to walk, they are familiar with remote controls, computers, cell phones, 
and other technology. All of which have changed the way we learn. To teach this group effectively, educators must 
keep abreast of developments in digital and web-based media and take advantage of the abundant opportunities they 
offer to help children learn. In addition, they must not only transfer knowledge to students but create "authentic tasks 
leading to participation in a community of practice" (Bonk & Cunningham, 1998, p. 26). 

Designing effective electronic learning environments may at times seem like a daunting task. Many hours 
of curriculum design and negotiating with our colleagues in technical departments can transform the learning 
experience for both students and teachers. When we take advantage of the many opportunities offered by online 
learning we inspire the new teachers in our courses to keep abreast of emerging technologies and choose appropriate 
activities to add to their own K-12 classrooms. The results can transform the learning experience for everyone in the 
learning continuum: professors, teachers, credential candidates and children. 

Professional teacher preparation programs, such as ours at National University, include a designed 
sequence of coursework and field experiences that effectively prepare candidates to teach students and to understand 
the contemporary conditions of schooling (National University School of Education, 2007). At National 
University the intent of the teacher education program is to prepare reflective teachers who can meet the needs of 
students in the diverse social and cultural environment in which they will teach. 

The aim of the course for this research. The Diverse Classroom, is to sensitize educators to the differing 
needs of students and to foster an exploration of students' beliefs and experiences with issues of diversity. Student 
diversity is defined as exceptional needs, including learning disabled, mentally retarded, behavior disorders, 
communicatively disabled, those with sensory disabilities, and gifted and talented. This diversity is viewed through 
the lenses of values, cultural differences, human development, and exceptional needs. 

The online version of The Diverse Classroom course was resigned as part of the university’s technology 
upgrades known as the Premier eLearning Project (PEP). The design of the course provides a solid foundation for 
relevant and purposeful online instructional activities. It offers an excellent paradigm for combining theory and 
practice in the teaching process. Salmon's (2000) five-stage e-moderating model and his E-tivities research (2002) 
provides a solid foundation for relevant and purposeful online instructional influence.. 

The course makes use of online elements such as threaded discussions, quizzes and lectures using flash and 
audio. Students are also required to complete traditional assignments, including individual course papers and 
fieldwork observations in K-12 classrooms. Among the updated features of the course is the use of an avatar to 
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introduce the course objectives and outline best practices for online learning success. The visual representation of 
the model is specifically designed to generate thoughts on ethnicity as the complexion morphs and changes into 
different faces. Written narratives give students a better perspective on the course content. It is never appropriate to 
tell students that they should change their way of thinking. The narratives on the podcasts provide students with 
new, deeper perceptions about the subject matter. The virtual tour brings a fresh look at the procedures teachers 
follow when working with exceptional needs students. The experiential learning simulations allow the participant to 
experience a measure of what it feels like having a sensory challenge. The interactivity within this course, coupled 
with fieldwork observations reports, create rich, authentic tasks for students that would not be possible in a more 
traditional learning environment. 

Data Collection 

To determine the student’s satisfaction with the technology features within the online course, a blend of 
quantitative and qualitative research methods were employed. Students enrolled in the January 2008 and February 
2008 course were selected for analysis. The end-of-course questionnaire used in this study was designed to generate 
attitudinal information from students and to examine the impact of the upgraded multimedia elements on student 
learning. The survey was administered by an online survey delivery platform where the respondent’s identification 
remained anonymous to the instructor and researcher. 

Initially there were 30 participants. Three students dropped the January section of the course after the first 
week. One student dropped the February course during the initial week leaving a total of 26 subjects; 8 males and 17 
females. Eighty one (81%) of the participants were single subject credential candidates, while nineteen (19%) were 
multiple subject credential candidates. They ranged in age, teaching experience and comfort level with web based 
technologies. 

Findings 

Among the participants surveyed, there was a feeling that the design of the course was a beneficial 
experience. Participants were satisfied with the amount and quality of the information presented. They were quite 
satisfied with the hands on experience with the audio, video and web links being the most popular elements. Overall, 
participants felt the course was a positive experience and they were departing with worth while content. 



Figure 1. Rate the level the online instruction provided a quality academic 

experience 
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Figure 2. The elements of the course learning content, instructional methods, 
technologies and course material complement each other. 
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Figure 3. Rate the level the following components enhanced your online experience 



Extremely Satisfied 
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■ Visual components 
(avatar;lectures) 

■ Web links 

■ Video 

■ Audio 



The following comments reveal the participants thoughts when asked what was most helpful in this course: 
“The interactive web sites. ””The amazing amount of information that was at our finger tips from textbooks, to web 
links and interactive. ””A11 the instruction by the instructor were easy to follow. I do not recall a signal time when I 
didn’t know what I was supposed to do.””The lectures.””! really enjoyed the multimedia links that showed the 
teachers classroom.””The multimedia.””The instructor she was always available for assistance.””! loved the various 
web sites such as the PBS site. The one on learners exceptionalities where I could experience firsthand the 
challenges was also wonderful.””Examples of course work the instructor was looking for from past students.” 

As evidence by the comments participants they were challenged by the course content and for the most part 
did not have difficulties with the technology that impeded learned. From the question inquiring what were the 
challenges in this course, the following comments emerged: “Arranging the observations. ’’’’Staying on top of work, 
required no procrastination and very organized work habits. I’m not always good at that so it was definitely a 
challenge. ’’’’Typos / duplicate questions on the quizzes.””For me it is different than younger students because just 
getting use to the computer was a challenge for me.””Getting the readings done. ’’’’Getting it to work on my 
MAC. ’’’’Writing to depth the teacher wanted. ””No challenges, great assignments that probed thinking and a lot of 
work.””The entire course was challenging to keep up with but spaced evenly that it was manageable.””The threaded 
discussions.” 
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Enthusiasm was expressed for suggesting the course to others in the future. On the question ‘Would you 
recommend others in taking this course? Five participants answered yes. Other written comments are as follows; 
“Yes, it was a great class. ’’’’Most definitely. ””1 00%. ””YES! It was great and I learned a lot.””Yes, the content was 
excellent. It made you think.” 

Significance of the Research 

As the range of multimedia features offered to online students increases there is a need to exam which 
elements are successful. We gain a better understanding of the learning processes, insights into difficulties, and 
develop strategies for responding. This study adds to the body of knowledge on how an avatar, podcasts, a virtual 
tour and multimedia experiential activities are used in an online class for adult students as well gives insights on the 
impact on learning. It makes the connections between new technologies and educational goals. Results from this 
study used have been used to improve the course templates and content for future students and faculty. 
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Introduction 

Through their socialization, individuals construct meanings reciprocally with the other members of the 
society and acquire skills and knowledge within this social interaction (Sivan, 2003). Different from traditional 
cognitive theories, according to social constructivist perspective, learning does not solely depend on individual 
cognitive activity but rather becomes a product of social engagement. Social constructivism not only serves as a way 
for cognitive development but also becomes a mean for individual’s interaction with the environment (Sivan, 1986). 

The traditional perspectives considered mentoring as a one-way process in which the guidance and teaching 
are provided by a mentor to a mentee. However from a social constructivist perspective, knowledge is co- 
constructed with interdependent social and individual processes (Palincsar, 2003). Having its roots from social 
constructivism, co-mentoring is defined as a collaborative learning process between the mentor and mentee. Based 
on this premise of the partnership in learning process, co-mentoring enhances learning through shared ideas and 
social construction of knowledge and skills. Both the mentor and the mentee act as learners and teachers in the co- 
mentoring process, which is based on “mutual trust and openness” (Mullen, 2000). Mullen (2000) calls co- 
mentoring process as “collaborative mentoring” in which each member directly entails in others learning and gives 
feedback “while developing along an agreed path” (p. 5). Sherry & Myers (1998) list the characteristics of co- 
mentoring: 

• Last for an extended period of time, 

• Based on each other's complementary knowledge and skills 

• Friendship of peers 

• No formal monitoring 

• Dialogue 

• Equal status 

• Reciprocal benefit 

• Being partners complementing each other's knowledge and skills 

• Mutual benefit 

This case study mainly attempts to investigate faculty technology experiences with social constructivist 
perspective and explore co-mentoring in technology learning and integration process. 

Research Methodology 

Case study is selected as a research methodology for this study as it allowed for extensive exploration of 
this technology mentoring experience. According to Yin (1984), “the case study allows an investigation to retain the 
holistic and meaningful characteristics of real-life events” (p. 13). It suits well the purpose of understanding 
collaborative mentoring and how learning technology from a social constructivist perspective operates in this 
particular case. 

Context and Participants 

Faculty Mentoring Program originated in the Department of Curriculum and Instruction at a Midwest 
University to meet the challenges that faculty experienced during technology integration to their curricular practices. 
Over time faculty who participated to the program have made significant advances in their adoption of technology 
and their ability to use technology to improve learning and teaching in their classes (Thompson, 2007). As part of 
the program, during the semester, graduate students (mentor) work with their faculty members (mentee) on different 
topics of technology integration based on their needs; meeting every week, learning and applying technologies. The 
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class hours also serve as a way to diffuse technology integration by providing a shared ground for mentoring 
experiences, new technologies and ideas on technology integration in teaching practices. 

Student Mentor Profile 

As a first year international PhD student in the program, my main aim was to be involved in this highly 
engaging practical experience with the support of my ongoing academic studies that the course offered. I had a 
background in “instructional technology” from my past academic studies and professional experience as an 
instructional designer. My major motivation for entering the faculty mentoring program was to combine my 
previous knowledge and experience in the field with faculty technology integration and thus gain hands on 
experience in instructional design and development. 

Faculty Profile 

My faculty mentee. Dr. Lee is an assistant professor at Curriculum and Instruction Technology Program 
whose major research interests include collaborative learning, design of instruction, and evaluation and assessment 
of educational products and interventions. She had extensive professional experience in teaching at both K12 and 
college level, working as an instructional designer and assistant professor at the University. She wanted to be a part 
of this mentoring program to use new technologies effectively in her teaching practices such as designing a graduate 
course on LMS (Learning Management System) and using a web conferencing tool as part of this course. 

Data Sources and Analysis 

Multiple resources were used for data collection: my mentor journal that was kept throughout the semester, 
artifacts that we developed and a face-to-face interview conducted with my mentee at the end of the semester. 
Qualitative data were analyzed to find common themes of critical factors in this collaborative mentoring experience. 
Interview transcripts were examined both for content and the metaphors that we used to describe our relationship. 
Journals were analyzed to determine the ways we interacted in our technology learning process. The themes for 
these analyses were categorized and coded. Social constructivist perspective provided an umbrella to explore the 
case and its connections with collaborative mentoring. 

Major Findings 

The following paragraphs discuss emergent categories identified during the data analysis. These contribute 
to a better understanding of co-mentoring approach evolved in this technology mentoring experience. 

Social and cultural factors that provided collaborative mentoring relationship 

During our technology mentoring experiences, social and cultural factors played important roles in 
developing our relationship and enhanced our collaboration. Among these are commonalities in our academic and 
professional backgrounds, cultures and the community that we belong to. Design was one of the common points in 
our academic and professional interests that provided this “common language” between us during the semester. 

Since both of us worked as instructional designers previously, we set a synergy at the first meeting. According to Dr. 
Lee, the similarities between our previous educational studies and professional works played an important role to 
initiate a productive communication. During our mentoring hours, the chemistry between us was also enhanced by 
many cultural factors such as our Mediterranean origin and social factors as belonging to the same community. We 
participated in the seminars, involved in the projects and they brought us together creating an engagement in 
addition to our mentoring practices. 

Evolving goals on shared ideas 

Dr. Lee set very clear goals at the very beginning and informed me about her needs of technology 
integration to her course. Instead of imposing these goals to me, she wanted to get my opinions about the integration 
of these technologies to her course and asked my recommendations regarding with semester plans. During the 
semester we used this approach of generating shared goals and expectations in our mentoring journey. 
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Designing based on shared negotiations 

Before designing a course on a LMS, we first tried to understand the structure of the platform and its 
modules that we could use in our course design. During the exploration process we shared the manuals, training 
materials and exemplary courses in the platform to generate some ideas about our design. During the design process 
rather than trying to impose our ideas to each other, we brainstormed, exchanged our opinions, generated potential 
solutions to our design problems, selected the best solution to meet our needs, implemented and tested the solutions. 
This discursive knowledge and opinion sharing helped us to develop shared understandings and act upon those 
negotiations. Below is an instance that I stated in my journal about our design process: 

We decided on designing an interface for the course that will reflect the idea behind it. We selected some 
photographs for the design of the draft interface. Later, we brainstormed on how to organize learning 
modules and course materials in a way that both the instructor and the students would easily use (Mentor 
journal entry). 

Exploring technology collaboratively 

Another technology that we experimented during the semester was a WEB conferencing platform. Dr. Lee 
wanted to integrate this platform to her course in order to set up an online meeting between the students in US and 
the students in Denmark. In one of our experiments we connected to the platform from different locations and tried 
to explore the features by experimenting it. This exploring process reduced our curiosity and anxiety about this new 
technology platform. During our experiments before the actual meeting, we noted down the problems that occurred 
due to technical and social issues and prepared a quick manual for users explaining the technical details about the 
online meeting. 

Building trust 

During our collaborative mentoring experience we built a relationship that based on mutual trust. Mutual 
trust made us feel comfortable during technology learning and experiencing process. We trusted each others’ 
problem solving skills and respected the decisions that were being made. 

Mentor as a partner 

Dr. Lee defines our mentoring relationship as partnership in which we both acted as partners in technology 
learning and integration process. Dr. Lee comments: 

With my mentor I don’t have to tell what I need, because she figured out without me talking, so it is like 
partnership or collaboration. 

In our co-mentoring relationship, instead of holding unequal positions, we developed a mutual relationship 
in which both of us were decision makers. The terms that we used at the beginning of the semester like “mentor” 
and “mentee” left their places to “learning partners” in our mentoring experience. Dr. Lee stated in the interview: 
Mentoring is so natural for me. . . it is part of my process, it is part of the struggles I am having this 
semester with technology, and sometimes my mentor is more as a partner on this learning adventure. . .1 
don’t have a clear distinction that she is my mentor. You are here because I am struggling and you are 
interested in this technology, but we develop together. 

Mutual benefits 

During our co-mentoring experience, we explored many collaborative learning technologies, tested them 
together and used them in Dr. Lee’s teaching settings. As a graduate student I experienced the teaching and learning 
settings by learning new tools for technology integration. Dr. Lee’s teaching and research experience guided me in 
my future works. Dr. Lee stated that she felt more comfortable using these technologies after our mentoring 
experience. 
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Discussion and Conclusion 



This case study showed that technology integration occurred through a collaborative learning model 
between the mentor and the mentee whose relationship was less a mentor-mentee relationship but more a 
collaborative partnership. In this collaborative partnership model, both the mentor and mentee contributed to the 
learning process with their different skills and competencies to solve a technology problem. Although the 
relationship between mentor and mentee was not defined clearly at the beginning, it evolved from traditional 
mentor-mentee relationship to a partnership model in which both parties learned and experienced technology 
together. Figure 1 demonstrates the elements of co-mentoring relationship that nurtured this faculty technology 
mentoring experience. 




Figure 2-Elements of co-mentoring relationship 

Researchers indicate that “mentoring is a dynamic and interactive experience” which can be enhanced 
through one-on-one mentoring models (Chuang & Schmidt, 2006). Unlike traditional models, providing students 
and faculty members with mentoring experiences that focus on shared sense of common goals, expertise and 
rewards may enhance the technology learning practices. Technology can play an important role in this respect, 
removing the hierarchy between the mentor and mentee and encouraging the open dialogue. This case study gave an 
example of how one-on-on mentoring is built on collaborative mentoring from a social constructivist perspective. 
Social constructivism as an umbrella to mentoring practices can provide valuable experiences both for graduate 
students and faculty members. In building this dynamic, reciprocal and nonhierarchical relationship, both the faculty 
and the graduate student may combine their experiences and knowledge to better integrate technology into their 
teaching and learning practices. 
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Introduction 

Asynchronous communication in online discussion environments presents new challenges to the design and 
organization of meaningful learning activities. Literature indicates several problems related to online discussion 
forums such as limited student participation (Hewitt, 2005) and limited critical analysis of peers’ ideas (Rourke & 
Anderson, 2002). To address some of those challenges, a number of facilitation strategies have been described in the 
literature mostly focusing on instructors’ facilitation roles (Anderson et ah, 2001; Paulsen, 1995). Although tutors or 
instructors play a critical role in online discussion environments, instructor dominated facilitation may result in 
instructor-centered discussion (Light et al., 2000) and limit students’ active participation (Pearson, 1999). Rovai 
(2007) claims that instructors “should avoid becoming the center of all discussions and encourage student-to-student 
dialog” (p. 83). 

This study explores successful facilitation strategies in online peer-facilitated discussions. If peer 
facilitation strategies or practices are able to promote meaningful dialogue and participation, then practical and 
theoretical implications can be drawn. From a constructivist perspective, online discussions may create opportunities 
for students to construct meanings together and integrate new knowledge into their prior experiences (Rourke & 
Anderson, 2002). Discussion platforms allow students and instructors to interact in social environments without the 
boundaries of time and distance, promotes students’ critical thinking and helps students to reflect on their ideas as 
they work at their own pace (Hew & Cheung, 2007; Wang, 2007). 

Peer facilitation does not hinder teaching presence but provides instructors a platform where they can jump 
into the discussions addressing misconceptions and helping students with their struggles (Rourke & Anderson, 2002) 
as well as sharing their own points of views, questions and challenges. In their study, Correia & Davis (2007) found 
that peer-facilitation in a discussion platform as opposed to the instructor-facilitation was the most successful 
collaboration design. Students found peer-facilitated discussions more meaningful and interactive where their 
participation created a strong sense of community (Correia & Davis, 2007). Analyzing post lengths and number of 
postings in an online discussion environment, Poole (2000) found significantly longer and higher number of posts 
during student moderation weeks. 

The adaptation of online discussion platforms in higher education has far outpaced our understanding of 
how this platform should be used to promote student engagement (Garrison et al., 2001). Despite the potential of 
peer-facilitation in a discussion forum, successful peer facilitation strategies that nurtures meaningful interaction and 
student engagement has not been widely explored in the literature and/or investigated. This is a key gap that the 
present study attempts to address. 
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The purpose of the study is to explore strategies used by student facilitators to promote peer participation in 
an asynchronous online discussion environment. Looking at both the quantity and the quality of the interactions, the 
authors opted to examine unique facilitation strategies employed by student facilitators to foster meaningful 
dialogue. 

Research Methodology 

An exploratory case study (Yin, 2003) was adopted in this study. A rich description of the peer interactions 
within an online course was completed with the intend to characterize successful student-led facilitation in 
asynchronous online discussions. 

Context 



This study takes place in the context of a master’s of education-at-a-distance in Curriculum and 
Instructional Technology program at a large Midwestern university in the United States. This program was primarily 
designed for K-12 teachers across rural Midwestern. This master’s is a three-year program consisting of 32 credits 
which is offered to a cohort of students every two years. Each cohort has 8 to 20 students, who are maintained as a 
group for the entire program. One of the courses in this program was an intermediate level instructional design class. 
Readings on instructional design principles, models, and strategies, were addressed in weekly synchronous 
discussions, which were led by students who volunteered every week. A limited set of facilitation guidelines were 
provided by the instructor and students were encouraged to explore different ways to promote meaningful dialogue 
and engage their peers in the discussions. 

Participants 

Sixteen students participated in this study in fall 2007 semester. Four of the students were male and 12 
were female, ranging in age from 22 to 55 years old. Thirteen of these students were part of the cohort pursuing the 
master’s of education-at-a-distance and the remaining three students were traditional master’s and doctoral students 
in curriculum and instructional technology taking this course as part of their program of studies. All students had a 
background on teaching and most of them were or used to be K-12 teachers. 

Data Sources 

Data collected included online documents such as: (1) course related materials (e.g., course assignments, 
students’ expectations); (b) instructions on asynchronous discussions (e.g., facilitator responsibilities); and student- 
led weekly discussion threads in instructional design issues. 

Data Analysis 

Three main approaches emerge from the literature for examining online interaction: quantitative 
approaches such as: thread length, number of postings; interaction patterns; and the quality of interaction (Nisbet, 
2004). In this study the analysis was a qualitative one of mainly online discussion threads to identify successful 
student-led facilitation strategies. A discussion thread is defined as “a hierarchically organized collection of notes in 
which all notes but one (the note that started the thread) are written as ‘replies’ to earlier notes” (Hewitt, 2005, p. 
568). 

In order to explore the extend to which student participation in an online discussion was successful, we the 
depth and content of the threads were analyzed. In this study a student facilitator was considered to successfully 
engage their peers’ participation in the discussion if the thread had a depth of at least six levels of messages. Both 
students and student facilitators’ messages were included in the depth count. Looking at a depth of six-level threads 
as the measure helped to see “if conversational exchanges or discussions are taking place” (Hew & Cheung, 2007, p. 
4) or whether a discussion is sustained or extended. Looking merely at the number of students’ postings was not an 
option in this study because such an approach would not represent an ongoing conversation. 

The qualitative data analysis consisted of on an iterative and inductive process of analysis. Through a 
careful analysis of the data, trends and discrepancies were found and categories emerged. The data analysis was 
done manually by reading and underlining, cutting, and pasting. Prolonged engagement, persistent observation of 
online activities and peer debriefing were used to support the credibility of the findings. 
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Findings and Conclusions 



Four mini-cases that included threads with a depth of six or more levels were found. In each mini-case a 
student moderated the discussion and used a unique and successful facilitation strategy to promote active 
participation from his or her peers. The richness of the conversation was also examined by considering the quality of 
messages related to the content. A preliminary analysis of these strategies and thread depth analysis is presented in 
the following paragraphs. To prevent the identification of the students, pseudonyms are used throughout this article. 

Mini-case 1 

The discussion was divided in two threads: one of the threads exhibited a depth of thirteen levels and the 
other achieved a depth of nine levels threads. All students contributed to the discussion. The facilitation strategy 
employed by one of the students, Cindy, was labeled as inspirational facilitation. 

The focus was on motivating her peers to imagine idealistic scenarios, search for inner goals and discuss 
ways to achieve them. In her role as the facilitator, she motivated and inspired the participants to share dreams, 
goals, and drives. As an example, she initiated the discussion with the questions: “If you had the opportunity to 
design a DREAM initiative, course, or classroom lesson, how would you go about implementing it at your 
institution?” Participants involved in the discussion shared their most daring dreams about change in schools. One 
discussion participant posted: “I enjoy this type of question because it let’s us dream! ! If I could have a dream 
classroom, I would envision personal handhelds or laptops for my students.” 

Mini-case 2 

Three out of seven threads got a depth of seven levels, one achieved a depth of twenty-one and another a 
depth of twenty-five levels threads. All students were involved in the discussions and contributed to threads with 
seven or more levels. The facilitation strategy employed here by the online facilitator, Molly, was labeled practice- 
oriented facilitation. 

The discussion is moderated around connections with real life situations, making frequent links to practice. 
By using this strategy, Molly encouraged participants to reflect on educational issues that related to their schools and 
classrooms. She posted: “On page 77, the authors thoroughly describe the instrumental process. Can you relate to 
this design process within your classroom or out?” Through this facilitation strategy participants used examples 
from their work, shared practitioner insights, and made connections between the readings and what they were doing 
in the field. One of the Molly’s classmates wrote: “Good question, Molly. In my first year of two years of teaching, I 
really tried to be this type of ‘instrumental’ teacher.” 

Mini-case 3 

In this case, of six threads, one thread achieved a depth of eight levels, another a depth of eleven and other 
a depth of twenty-three levels threads. All group members contributed to threads that achieved a depth of seven or 
more levels. The facilitation strategy is defined here as highly structured facilitation and was employed by David. 

This strategy aimed at supporting facilitation by offering an organized process that kept the discussion 
focus on the issues at hand. For instance, David moderated the discussion in a systematic way by using the K-W-L 
method. He asked his peers to explore the readings by answering the questions: “What do you want to Know?” (K), 
“What do you Want to know?” (W), and “What did you Learn?” (L). He managed the discussion by methodically 
responding to his peers’ postings and providing resources. David’s peers found this strategy very helpful in leading 
an online discussion. They requested resources about this strategy and one of his classmates commented: “This is an 
excellent strategy to use with students, so thanks for allowing us to practice it as well!” 

Mini-case 4 

In this last case, of two threads one of them achieved a depth of six levels and other a depth of fourteen 
levels threads. The facilitation strategy was a community-driven facilitation used by Laura. 

This facilitation strategy focused on bringing the students together by building on a sense of community. 
Laura gently invited responses, asked clarification questions and acknowledged contributions. Here is one of her 
postings: “Nice point, Kim - having a great curriculum like yours, Courtney, seems to free you from needing to 
write objectives and enable you to focus instead in the diverse needs of your learners!” She was able to bring 
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everyone together into the discussion and create a risk-free discussion. One of her classmates confessed: “I don’t 
usually write objectives (or formalized lesson plans for that matter) unless I am being observed.” 

Discussion & Conclusion 

Much of the research dealing with facilitation emphasizes the importance of teacher’s presence in online 
learning environments (Hara et al., 2000; Zhu, 1998). Although teachers’ moderation has been considered essential 
by a number of researchers (Anderson et al., 2001; Berge, 1995; Paulsen, 1995), other studies indicate some of the 
shortcomings of instructor-dominated facilitation in online discussions (Seo, 2007). For instance, instructors may 
not be able to fulfill all of their moderation responsibilities because facilitating an effective online discussion 
requires time (Rourke &Anderson, 2002) and dedication. Consequently, managing large discussion groups may be 
overwhelming on the instructor’s side (Seo, 2007). Moreover, instructor moderation may lead to instructor-centered 
discussion (Light et al., 2000) creating an “authoritarian presence that the instructor brings to the discussion” 
(Rourke & Anderson, 2002, p. 4). 

In order to overcome the challenges of an instructor-dominated facilitation, enhance the sense of learning 
community and encourage students’ participation, the study described used peer-facilitation for online discussions. 
This study showed that students use successful facilitation strategies to enrich interaction and guarantee active 
participation among their peers. Research on students as online discussion facilitators also suggests that this 
approach is beneficial for both student involvement and learning outcomes (e.g. Rourke & Anderson, 2002) having 
the potential to reduce the instructor’s related workload while teaching online. Using successful facilitation 
strategies, students explore new ways to engage their peers in an online discussion. 

Preliminary findings in this study indicated that the four facilitation strategies generated innovative ideas, 
motivated students to participate in the discussion in different ways and provided a risk-free and relaxed atmosphere 
for discussion. According to Tagg (1994), a “direction from within” approach requires a reconsideration of 
facilitation roles, which are traditionally linked to leadership. This change of roles means giving students the power 
to take practical and meaningful roles in the online classroom. As such, by analyzing both the quantity and the 
quality of the discussion threads, it was found that this “direction from within” approach served as an empowering 
opportunity for the students. 
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Introduction 

Training departments in large organizations are typically tasked with providing hundreds of hours of 
training content to several thousand employees each year. With often short duration development timelines and 
multiple internal and external (vendor) content developers simultaneously developing several projects for several 
stakeholders, the logistics of managing these development efforts can be daunting. These organizations are 
increasingly turning to Learning Content Management Systems (LCMSs) to centralize the creation and management 
of training content (Hall & Hall, 2004). 



What is an LCMS? 

International Data Corporation defines an LCMS as a system “used to create, store, assemble, and deliver 
personalized e-learning content in the form of learning objects" (Oaks, 2002, p. 73). The primary purpose of an 
LCMS is to manage and streamline the content/course development process. Using the Advanced Distributed 
Learning (ADL) Sharable Content Object Reference Model (SCORM) (Advanced Distributed Learning, 2008), an 
LCMS will allow a distributed team of developers to reuse or reconfigure existing content and use the same content 
to create multiple courses for multiple audiences (Hall & Hall, 2004). Additionally, an LCMS automates course 
publishing and updating, and facilitates version control of dynamic training content (Hall & Hall). 

Perceived LCMS Benefit 

A 2006 Bersin Associates report stated that 28% of managers involved in training development were using 
some type of an LCMS, and 42% stated the intention to acquire an LCMS in the future (Howard, 2007). The stated 
purposes for either using or considering an LCMS included the need for rapid content development, automated 
content development, and managing large libraries of constantly changing content (Howard). The LCMS addresses 
these needs by providing for the management of: 

content in a highly accessible, automated database where the work of untold legions of training 
professionals is combined into one centralized hub, those who need to design a course from scratch, build a 
new course using existing content, or find chunks of knowledge or learning objects to plug into a course 
under development can do so in a flash (Hall & Hall, 2004, p. 33). 

Other perceived benefits include the ability to tie the LCMS into a learning management system (LMS) and 
deliver customized content to learners. For example - a learner searches the LMS for a course and is presented with 
customized modules from various e-learning content. The learner can either complete all of the modules, or 
complete relevant modules tagged from the LCMS. 

Potential LCMS Issues 

In a large organization, the selection and implementation of an LCMS is not done solely in the silo of a 
training department. Multiple stakeholders; such as Information Technology, Finance, Human Resources, and even 
Operations can, and often will, be involved in the selection and implementation process. Another aspect of very 
large organizations is that there is often not a single unified training department but instead several smaller training 
departments, each serving different business units. As a result, the LCMS selected, as well as the implementation, 
may not suit the perceived needs of all stakeholders, especially the stakeholders of the dispersed training 
departments. 
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An additional issue surrounding content development in an LCMS environment is the fidelity of the content 
the LCMS is capable of producing. Aldrich (2003) described two categories of LCMS content output, workbook- 
style web pages and virtual classroom sessions. Aldrich went on to say, “One of the most disturbing aspects of the e- 
learning movement has been the lack of focus on the quality of end-learner content” (p. 60). Many e-leaming 
developers see the LCMS ‘workbook-style web page’ output to be a contributing factor to the perceived lack of 
quality with many large organization e-learning programs. 

Another potential issue is how instructional design, instructional strategy selection, learning domains, and 
the LCMS intersect. The LCMS has quite specific functionality and depending upon the business rules of the 
organization, instructional designers or instructional media personnel may be ‘hog tied’ into a solution. The potential 
issue can be summarized as the technology driving the instructional strategy - or the tail wagging the dog. 

A Case Study 

It is important to understand that the selection and implementation of an LCMS is a significant financial 
investment, and fundamentally changes the way organizations develop and deliver training. Many stakeholders 
involved in the selection and implementation invest significant political capital in the process and often careers 
hinge on the success or failure of the initiative. For this reason, the authors of this paper have chosen not to identify 
the players (company, vendor, and LCMS platform) in this case study. 

In late 2007, a global Fortune 500 company (hereafter referred to as the company) collaborated with a 
leading learning solutions provider (hereafter referred to as the vendor) to develop content within the newly 
implemented enterprise LCMS environment. Prior to the LCMS implementation, the company and vendor had 
collaborated to develop custom content, which was perceived within the larger organization as best-of-breed, and 
received two prestigious e-learning industry awards. With this background of past excellence, the vendor’s initial 
development of content within the LCMS was received by the organizations’ stakeholders with disappointment 
(‘shock and awe $!&#’ might be a better description). 

Instructional Designers in Bondage 

Jerome S. Bruner (1960, 1966) suggested that intellectual ability is developed in stages through step-by- 
step changes in how the mind is used. Bruner also suggested through his theory of cognitive growth, that when 
developing learning it should be based on environmental and experiential factors. - learning should be based on a 
‘spiral’ notion - starting with the basic information (iconic or symbolic) and move toward direct purposeful 
experiences (enacted). Edgar Dale’s well-known Cone of Experience (1946) supports this also. Dale’s Cone of 
Experience aids instructional designers in selecting media based on the experiences - active learning; experiences 
are not completely relied on visual or verbal symbols - passive learning. Within Bruner’s and Dale’s theoretical 
frameworks, the ability for instructional designers to choose from a wide variety of instructional strategies is 
paramount. 

One of the primary causes of the disappointment was that the ‘workbook- style web page’ output of the 
LCMS (template-ized web pages) did not meet the expectations of fidelity the stakeholders experienced in previous 
non-LCMS development efforts. The templates available in the LCMS environment resembled little more than 
generic and elementary PowerPoint presentation slides. Eeldstein (2002) suggests that LCMS users avoid the 
common LCMS gimmick of importing PowerPoint presentations into the LCMS by asking "why not just distribute 
the slides and be done with it" (p. 4)? This raises the question that if importing PowerPoint presentations into an 
LCMS is a bad idea, then would not creating content within templates that resemble PowerPoint presentations also 
be a bad idea? 

Eeldstein (2002) makes three keen observations regarding what aspects that we should keep in mind when 
developing good e-learning: 

1 . A broad range of presentation styles is required 

2. Learning should be interactive 

3. Good learning content is difficult to write 

Additionally, Eeldstein cautions that the LCMS should not “not prevent course designers from using the same tools 
and techniques they have always used. In particular, it should not make it harder for web programmers to add 
custom code” (p. 2). 

The rigid and limited number of templates available in the LCMS did not provide for a broad range of 
presentation styles and allowed for limited interactivity. It also limited creativity in the learning solution. This 
drawback could have been over come with custom code, but the business goal to realize course development costs 
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savings promised by migrating to an LCMS environment precluded including custom code in the development 
budget. Additionally, any custom code would have needed to be specific for the LCMS environment. This would in 
effect be LCMS platform unique code which is typically more expensive, difficult (by virtue of the learning curve) 
to develop, and not easily migrated to other delivery platforms. 

Providing interactivity was hampered further by technical considerations that limited the web-browsers that 
the few interactive templates would function in, and the requirement that the interactive templates be delivered to 
the end user (learner) through the LCMS itself. As the company is a global organization with several hundred 
thousand employees world wide, restricting the training to only a single web-browser version and requiring all 
learners to access the training through the same portal was not practical. As a result, the more interactive templates 
and features were not usable. 

Another key limitation was the assessment and knowledge check functionality of the LCMS. When 
measuring knowledge transfer or skill obtainment for more complicated learning objectives the use of simple 
multiple-choice, true-false, or matching items may not be sufficient. Additionally, it is instructionally appropriate to 
provide feedback for both correct and incorrect answers so that the learner does not leave with the impression of the 
incorrect answer being correct. The LCMS limited the available assessment options. This posed a challenge for the 
instructional designers in selecting the appropriate assessment type. Additionally, feedback and prescriptive learning 
functionality was limited. In some cases the feedback had to be all on or all off - meaning, you cannot turn the 
feedback on if the learner passes the assessment so that the learner can see any missed items, and you cannot turn it 
off if the learner fails and you want to send him or her through a prescriptive learning activity. There were also are 
challenges in creating multiple attempts with prompts for an assessment item. 

These limitations shackled the instructional design team. As Feldstein (2002) points out. Good learning 
content is difficult to write, and it is even harder when the designer’s options for presentation strategy and 
interactivity are severely limited. It is important to keep in mind however, that it is not only technical considerations 
drive the limitations. The perceived business need for need for rapid, automated content development also drives the 
LCMS to shackle the instructional designer with a simplistic, template environment - but is it worth it? Feldstein 
states, “ease of use and reduced labor are important, but they are not terribly helpful in the end if you have to 
sacrifice the ability to create quality courses in order to get them” (p. 2). We would suggest that the business 
considerations go beyond “not terribly helpful” but instead result in pedagogically unsound training practices that 
may in the end cost the organization more in lost productivity than was saved in the learning development. 

The Myth of Object Reusability 

One of the LCMS features which organizations see as key to helping reduce development time and costs is 
the concept of object reusability. In theory, attaching SCORM metadata to objects (chunks of text, media, course 
screens, lessons, or even entire courses) would prove to be the holy grail which would enable course developers to 
as Hall and Hall (2004, p. 33) put it “ find chunks of knowledge or learning objects to plug into a course under 
development ... in a flash.” A 2005 report by Howard (cited by Wilkins, 2007) suggests however, that while 77% of 
organizations cite object reusability as their primary reason for implementing an LCMS, only 38% are actually 
reusing content and then only at a reduced scale. In the case of the company, object reusability was indeed one of the 
primary reasons for implementing the LCMS, but our experience quickly showed that object reusability is at best 
highly problematic, and at worst as Wilkins suggested, a myth. The reasons that prevented object reusability in this 
case were both pedagogical and technical. 

From a pedagogical perspective, it is helpful to remember Feldstein’ s (2002) assertion that good learning 
content is difficult to write. Many factors contribute to this difficulty, not the least of which is that good learning 
content must address the stated learning objectives. A quick review of the stated learning objectives from multiple 
courses within an organization’s offerings will likely not reveal the same objectives repeated in multiple courses. 
After all, if the objectives never change, why would we need to develop multiple courses? Therefore, while 
instructional designers realize that good learning content is unique to the learning objective, the concept of content 
reusability does not make this distinction and assumes that learning objects are interchangeable parts. This reduces 
learning design and development to a manufacturing process of which we are sure Henry Ford would approve, but 
would Plato? This is not to say that reusability always inappropriate. Instead we suggests that the whole sale object 
reusability which is one of the primary selling features of the LCMS is counter intuitive to pedagogical best practice. 

Without considering pedagogical issues, certain technical issues also limited the realization of object 
reusability. The first issue we encountered was unique to the organization, but we feel not so unique as that other 
large organizations may not have the same experience. In order for all learning objects to be reusable, all developers 
must be sharing the same development database. In our case, security and licensing issues prevented the vendors 
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(three in total) from sharing the same instance of the database. In effect, four organizations were developing content 
for the company; the company itself and three vendors. Though each development team may be able to identify and 
reuse an object within their own instance, there was no ability for all database instances to share objects so each 
organization was limited to the reuse of the objects they created only. 

Another issue surrounding reusable objects is that if the object is changed in any way that change is 
automatically migrated to each instance (course) in which that object is used. This functionality directly relates to 
the argument that just as learning objectives are unique, so are the objects that support them. If a used object is 
changed, a mechanism must be in place to ensure that the change is appropriate for all instances. Because of the size 
of the company and the number of courses that were being created and planed (literally hundreds across multiple 
business units) the logistics of validating the appropriateness learning object changes were deemed insurmountable; 
thus object reusability, one of the primary LCMS business drivers, was abandoned. 

Finally, achieving object reusability became a labor cost issue. In order for an object to be reused, other 
developers must be able to find it. This is accomplished by embedding metadata, which allows other developers to 
find the object using key word searches, into the object media file. The problem becomes, what are the key words? 
For example, when developing training for tech support personnel, the developer may tag a picture of a help desk 
technician on the phone with the metadata tags help desk or technical support. This image of course would be 
appropriate for call center training also, but with the previously mentioned tags searching for call center or call 
center representative would not return the image. To overcome this, instructional designers (the most expensive 
labor asset on the development team) found themselves spending significant time thinking of all the potential 
metadata tag possibilities. Though adding extensive descriptive metadata to each object sounds like a good idea, 
once several thousand objects have been added to the data base (and remember everything including text on screen 
is an object), doing a metadata search on a common term, such as policy, can produce returns that make a Google 
search seem tame. After multiple trials and errors with the metadata, it was decided metadata reflecting the screen 
title and the course number, was necessary to a least allow developers to easily find and change an object in a 
course. This of course made the metadata course specific. In effect, this decision was the final nail in our object 
reusability coffin. 



Our Solution 

We hesitant to say that we solved the problems we faced with the LCMS. A more accurate statement would 
be to say that we worked around the issues. As we have discussed, the issues regarding content reusability were not 
truly solved, but instead we in effect abandoned the concept. This is disappointing because object reusability was 
one of the primary reasons for implementing the LCMS. 

The lack of object reusability was not a showstopper for us, after all, we had never had it before and 
reusability is not a requirement for good e-learning. The ‘workbook-style Web page’ LCMS output with limited 
interactivity however, was a showstopper, especially when some of the companies’ pre-LMCS e-learning was 
considered among the best in the industry. To over come this, we went back to one of the tried and true custom e- 
learning development tools, Adobe Flash. By creating custom Flash animations and interactive objects, and 
embedding those objects into the learning content white space provided by the LCMS, we were able to raise the 
quality bar to a level approaching that of our pre-LCMS development efforts. In doing this however, the 
development time and costs savings promised by the LCMS were not realized. 
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Introduction 

One of the most promising benefits of networked technologies is the ability to support collaborative 
learning experiences among learners separated by geographical boundaries, thus allow the creation of online 
learning communities across borders. Research in the area of social constructivism has revealed that collaborative 
learning allows learners to maximize their own and each other’s learning (Johnson & Johnson, 2003, p. 488). 
Collaborative projects, as the one described in this paper, required a careful prioritization of learning resources, 
experiences and practices, which would support the development of a group identity. This happened mainly because 
of the complexity of the elements involved, such as, language issues, diverse backgrounds, cultural differences, 
technical difficulties and time differences. About this complexity in relation to the learning process, Wenger (1998) 
states; 

Learning (...) takes place through our engagement in actions and interactions, but it embeds this 
engagement in culture and history. Through these local actions and interactions, learning reproduces and 
transforms the social structure in which it takes place (...). Learning is the vehicle for the evolution of 
practices and the inclusion of newcomers while also (and through the same process) the vehicle for the 
development and transformation of identities. (p.l3) 

Collaborative learning experiences expose learners to a variety of ideas, perspectives, and approaches to 
problem-solving activities (Johnson & Johnson, 2003). A virtual team is a working or learning team that functions 
without team members being present in the same place or even at the same time and uses mainly technology to 
support its work. Also, team members on virtual teams often cross many borders — national, organizational, and 
functional — and other boundaries such as, social class, race, age group, gender, economic status, religion, sexual 
orientation, professional backgrounds, ethnicity, lifestyle, knowledge, and skills. In this study, culture is at the heart 
of the design of experiences, and the multitudes of differences evolve into shared instructional design solutions. 

Being part of cross-border virtual team allows for cognitive disequilibrium, which is conducive to learning, 
creativity, cognitive and social development. Additionally, collaborative experiences lead to an increase of 
sophistication in elaborating arguments, sense-making, position-taking and consensus reaching (Johnson & Johnson, 
2003). 
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Online social presence, the fundamental element for the creation of a collaborative knowledge building 
process through what Wenger (1998), calls “negotiation of meaning,” is threatened by the lack of both 
“participation” (interaction) and ’’engagement” (Gunawardena, 1995; Rourke, Anderson, Garrison & Archer, 1999). 

This seems especially truth when collaborative learning involves interaction among groups of people 
coming from diverse backgrounds, especially ones defined by nationality. This situation adds one extra layer of 
complexity to communication; the intercultural element that affects the way people solve problems and achieve 
agreement among each other. 

This paper describes an online collaborative learning experience between graduate students from two 
higher education institutions located in the U.S. and Denmark. The purpose of this case study was to understand 
collaborative learning in the context of cross-border virtual teams. 

Research Methodology 

Case study research was chosen as the methodology for this study as it allowed for a deep exploration of 
cross-border virtual collaborative learning experiences. According to Yin (2003), case study allows for a holistic 
examination of a phenomenon within real-life contexts. The following paragraphs describe the case study of a 
virtual collaboration between eleven graduate students pursuing advanced degrees in educational technology in the 
U.S. and Denmark. The study took place from November to December of 2006. 

Contexts 



Located in the Midwestern of the United States, the Lands College is part of a large research university, 
known by excelling in science and technology applications. The College is comprised of six different academic 
programs and departments ranging from curriculum and instruction to apparel design and production. The graduate 
program in curriculum and instructional technology is housed in the curriculum and instruction department and aims 
at providing “leadership in the use of information technology in teacher education through research, development 
and service within a pluralistic and global society.” The department also offers a technology-rich and synergetic 
environment organized around a center for technology in learning and teaching. 

The Ankor University is located in Denmark and has an excellent reputation in its interdisciplinary 
problem-oriented graduate programs. The communication department at Ankor University houses the online master 
in information and communication technologies in learning. This program is the result of a collaborative effort 
between Ankor University and other four highly regarded higher education institutions in Denmark. The program 
aims at using information and communication technologies to “develop new learning forms to master all degrees of 
‘the learning society’ and life-long education.” 

As a result of an on-going teaching and research collaboration between these two universities and 
departments (Correia & Sorensen, 2007) graduate students from both programs were invited to participate in this 
case study. Students who volunteered to participate were assigned to mixed teams and worked on an instructional 
project for a 4-week period. This virtual learning and teaching experience was collaboratively designed, developed, 
delivered, and evaluated by the course instructors from both universities. 

Participants 

Of the eleven graduate students who participated in this case study, 5 were female and 6 were male. Their 
ages ranged from 26 to 50 years old. In the U.S. program, one student was originally from the U.S. and the 
remaining four were from Turkey, India, and Ukraine. In the Danish program all students were originally from 
Denmark. Most of the students had extensive professional and teamwork experience and were proficient in using 
technology. The students were organized in three teams according to general criterion of heterogeneity. For instance, 
teams had representation from both programs (U.S. and Danish), and showed different levels of virtual teamwork 
experience, background, and professional experiences. 

Project- related tasks and technologies 

The main drive for the collaborative learning experience was a short-term project that consisted of 
brainstorming and identifying through consensus a world-wide educational problem or issue. The main objectives 
for the project were: (1) to enhance learning through technology and collaboration and (2) to practice consensus 
reaching in virtual cross-border teams. The project included the following tasks: 
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— Participate on a kick-off meeting using a web-based video conferencing tool. The major purpose of this 
meeting was to socialize and to get to know each other better. 

— Get started on a learning management system (WebCT). All students had access to this virtual environment. 

— Get started on virtual teamwork using resources available in WebCT. 

— Brainstorm a world-wide learning problem with their team members. A problem that if solved would help 
people to learn better. 

— Reach consensus on the problem to address. 

— As the closing task for the project, teams had to prepare a presentation to share with the large group the 
educational issue identified and the team processes used via web-based videoconferencing. Students would 
also be responsible for this virtual session full planning (scheduling, agenda, activities, sequence and 
overall strategy). However, the virtual meeting should not go beyond 1.5 hour, giving each team 30 
minutes for their presentation. 

The main types of technologies used for online synchronous and asynchronous communication were: (a) the 
learning management and delivery system, WebCT', and (b) a web-based videoconferencing tool. Acrobat Connect 
Professional. The interaction between students was not limited to these settings; instead they were encouraged to use 
other online communication platforms to further support the collaboration (e.g., Skype and Google Docs). 

Data collection 

The data was collected through in-depth semi-structured interviews which focused on participants’ own 
expressions about the experience and provided opportunities for asking further questions and clarifying meanings 
(McCrachen, 1988). Site interviews were conducted with individual team members in U.S. Acrobat Connect 
Professional was used to conduct individual interviews with the Danish counterparts. This videoconferencing 
system was also used as a research tool in addition to a synchronous computer-mediated communication platform. 

Data analysis 

Interview transcripts were examined for content. Emergent themes centered on team members’ critical 
experiences were categorized and coded. To support the credibility of the findings researchers first analyzed the data 
independently and then discussed their interpretation to reach a consensus on the emergent categories. 



Major Findings 

The following paragraphs discuss emergent categories identified during the data analysis. These contribute 
to a better understanding of collaborative learning in the context of cross-border virtual teams. This section presents 
summaries of interviews and the authors’ interpretations. 

The nature of the task: social needs versus task requirements 

Because of the many borders being crossed, participants felt an acute need to share their personality traits, 
life experiences, expectations, motivations, values, and interest. The time spent together in virtual spaces was used, 
especially in the beginning of the project, for more of a social purpose than a task-oriented one. The project was the 
“excuse” for coming together but not necessarily always the main focus of the collaboration. Team members 
engaged in conversations outside the project’s scope but critical for its success. One of the participants in this study 
commented: “If you are communicating with a person face-to-face, there are many other things with which you can 
relate to a person; but if you are communicating virtually. . .online, you don’t have anything to hold on. So if you 
share your personal things that make a lot of difference in the communication.” Adult learners require social 
activities as strategies to build community. Problem-solving is impacted by the social conditions in which 
individuals are entrenched (Ahern et ak, 2006). Devaluing social practices when dealing with adult learners 
decreases the changes of success of any educational enterprise. In this same line, one of the participants mentioned: 
“Our first meeting in Skype was to talk about our personal experiences. It was my idea and I think it was good 
because we could pull something out from our experiences and interests and make a contribution to the project.” 
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The nature of the social relationship: can you build trust immediately? 

Participants agreed that a serious obstacle for establishing and maintaining a virtual social relationships was 
the lack of nonverbal information and contextual cues, even when they were able to interact via teleconferencing. 
One of the participants explained: “Because you don’t get any visual cues really, you don’t see people. Even if you 
do seem through like a webcam, you don’t make an eye contact. They can’t tell if you are looking at them or they 
can’t see you. So you do not get reciprocal relationship. . . It is so difficult when you can’t look around the room, you 
can’t tell when you are loosing people. . .” Cross-border virtual teams tend to be temporary teams, meaning team 
members do not have a history of working together. As a consequence social relationships evolve differently. It has 
been argued that these teams develop a different type of trust: swift or fast trust (Blomqvist, 2005; Meyerson, Weick 
& Kramer, 1996). To allow for effective decision-making temporary teams require trust to be quickly established 
and maintained without face-to-face interaction (Mezgar, 2005). When asked if his team has developed a trustful 
relationship, one of the participants commented: “Actually we were ambitious in the start, we were trying to discuss 
problems very fast, perhaps it would be better if we took more time knowing about each other, a little bit more time 
to talk about what Y. was doing, where she lived and so on.” 

The virtual collaborative learning experience: different starting points. . . and time differences 

Most of the participants complained different starting points among students in U.S. and in Denmark in 
respect to the project main objectives. In cross-border virtual teams it is important to make sure that every team 
member clearly understands the goal and objectives of the project despite their cultural, academic and language 
differences. Starting from a common point can make the negotiation process less painful and lead to a more fruitful 
experience. That is the reason why it is essential for successful teams to have clarity of purpose. Clearly stated team 
work requirements could give team members a shared starting point and provide a work structure to build on, and a 
defined work flow. 

One other issue identified was a 7-hour time difference between U.S. and Denmark. Such a situation made 
difficult to schedule synchronous meetings and created some confusion and misunderstandings. One of the 
participants reflected on a related episode: “But I completely lost the sense of time. They were 7 hours ahead. And 
the time I suggested for the meeting on Skype was midnight over there in Denmark. How stupid am I to tell 
someone to meet with me at midnight? I didn’t realize that and later I wrote T am sorry, I guess time difference is a 
big thing which we have to work around.’” 

Content: educational issues do not have frontiers 

Participants realized that when it comes to education many of the issues are universal. Technology 
integration, teacher training, and online assessments were the topics addressed by the teams. They found that when 
discussing educational issues there were more commonalities than differences among them. Teams took their 
different perspectives and integrated them into a coherent approach. They learned how to leverage their differences 
into common solutions. One of the participants commented: “Our project was to come up with new ideas on how to 
assess students online. . . And it was interesting to see how we came up with this idea. Because we were teachers, E. 
in Denmark and me here. We had teaching background, experiences and were dealing with similar issues, 
standardized assessment, and we were both disappointed with this kind of assessment.” 

Technology: used as a transparent medium 

Despite the initial technical challenges, time differences, and geographical locations, the technology, 
especially after the project kick-off, began to be used as a transparent medium. It allowed teams to focus on the 
process of collaboration and not so much on the types of technologies. Web-based videoconferencing systems, in 
particular, were successfully used to support teamwork because of the capabilities of supporting multiple users and 
format (audio, video, and documents). This allowed team members to communicate virtually in conditions that 
mimicked face-to-face interactions. As one participant explained: “Chatting is a good way of talking, because you 
can get a transcription of the chat, but it’s also demanding because of the differences in our language. It’s hard to 
write down our thoughts in a chat, because the answer is expected to come at once and quickly. And it can be 
difficult to keep one thread because of the speed in which you can reply [sic].” 
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Discussion & Conclusion 



In conclusion, students and instructors realized that it was critical to (1) design collaborative experiences 
that meet social needs, (2) pay close attention to issues such as, listening skills, communication styles, consensus 
reaching strategies, and language issues, and (3) be straightforward about the problems and barriers faced (e.g., 
technology failure) sharing frustrations and disappointments. Feeling part of a community that quickly develops 
trust and greatly invests on social activities is one important step for successful collaborative learning experiences 
with cross-border virtual teams (Correia, Baran & Yusop, 2007). One of the critical factors of success is to take time 
and create opportunities for sharing personalities traits, life experiences, expectations, levels of commitments, 
extrinsic and intrinsic motivations, perspectives for learning and teaching, etc. within each participant’s limits of 
comfort. Designing collaborative learning experiences that span time, geographical frontiers, and cultures can be 
challenging, but with carefully planning from initiation to completion these experiences can be rewarding to 
participants as well as conduct to creative outcomes. 

What would be useful for instructors to know in order to promote identical collaborative experiences? 

— Design collaborative activities to meet social needs (e.g., games on the importance of cultural values, i.e., 
time, work, education, ethics, and justice). 

— Provide time and suggest technologies for teams to use to get acquainted with each other. 

— Define a clear project goal and objectives and make sure everyone understand them before the start of the 
project. 

— Engage participants as much as possible on the running of the collaborative experiences. It creates a sense 
of commitment and ownership, which are critical for fruitful collaborations. 

— Ensure that most of the participants are familiar with the types of technology used to support teamwork. 
Otherwise participants are going to put most of their energy into learning how to use these technologies rather 
than engaging in the collaborative activities. 

— Be straightforward about the problems and barriers faced (e.g., technology failure, language issues, 
communication styles), sharing frustrations and disappointments. This way participants understand that these 
challenges are a common issue and do not “only happen to them.” 

— Acquire an in-depth understanding of the types of technologies to be used, which requires pushing 
technology to its limit by leveraging limitations and expanding capabilities. 
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Introduction 

In an increasingly interdependent world, companies not only have suppliers and customers around the 
world, but they also operate in many countries, creating multiple teams across divisions and organizations, which 
use technology as their primary mode of communication. Simultaneously, at the university level, students engage in 
experiences as members of virtual teams getting prepared to communicate across cultural and organizational 
boundaries using technology. Learning while working together is becoming mandatory to meet workplace 
performance requirements, and students need to have authentic experiences in this area while earning a college 
degree. However, the challenge appears to be in designing and supporting high-performance teams as well as in 
using technology effectively to facilitate teamwork. 

Since trust has been identified as the defining issue in understanding the effectiveness of virtual teams 
(Aubert & Kelsey, 2003; Poole, 1999), instructors and team members might benefit from understanding trust 
formation in virtual learning environments. However, there is a clear lack of research focusing on trust in virtual 
learning environments since most of the trust research is centered on the relationship between supervisors and 
subordinates (Schoorman, Mayer & Davis, 2007). 

This study focuses on the relationship between trust critical elements and trust outcomes in the context of 
virtual learning teams. Trust development in virtual teams represents a significant change in group dynamics, 
especially when team members can rarely meet face-to-face. A confirmatory factor analysis conducted aimed at 
demonstrating the viability of the theoretical framework proposed to explain trust building in virtual learning teams. 

Theoretical Framework 

Teamwork is frequently part of academic work because it is viewed as important preparation of students for 
real-world implementation. The complexities and expectations of learning groups have grown; yet, we still 
understand little about the inner workings of these groups (Schoorman et ak, 2007). A virtual learning team is 
defined here as a group of people who learn together largely through computer-mediated communication rather than 
face-to-face interaction. 
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Trust and Team Performance in Virtual Learning Teams 

In this proposal, Mayer, Davis and Schoorman’s (1995) definition of trust was used. Trust is “the 
willingness of a party to be vulnerable to the actions of another party based on the expectation that the other will 
perform a particular action important to the trustor, irrespective of the ability to monitor or control that other party” 
(p. 712). 

Based on an analysis of theoretical approaches to explain the phenomenon of trust development in virtual 
and face-to-face teams, a theoretical framework of trust building in virtual learning teams is proposed (Figure 1). 
This framework is partially based on the conceptualization of antecedents of trust by Mayer et al. (1995) and Aubert 
and Kesley’s (2003) research on trust and performance in virtual teams. It is argued that three elements are critical to 
facilitate trust building in virtual learning teams (antecedents of trust): 

1 . Ability, which recognizes that team members have the competencies, adequate knowledge and skills, and 
the qualities for the task. 

2. Benevolence, meaning consideration of others, esteem, and desire to do good towards one’s fellow team 
members. 

3. Integrity, which consists on the firm adherence to a set of principles that all team members accept. 

Learners’ satisfaction with team products and processes are identified in this framework as critical factors 

for overall team performance (outcomes of trust building). Teamwork processes are series of activities conducive to: 
(1) efficiency and effectiveness of processes that create desired effects (e.g., good grades); (2) team members’ 
professional growth; and (3) quality of interpersonal relationships. Teamwork products are represented by the 
quality of team outcomes reflected in innovation of products, services, or ideas. 




Figure 1 - Theoretical framework of trust building in virtual learning teams. 



Research Methodology 

Bagozzi et al. (1979) explain that confirmatory factor analysis using the structural equation modeling 
technique is a rigorous method for determining the convergent validity of an instrument as it provides a goodness- 
of-fit test for the proposed models. Therefore, confirmatory factor analysis and path analysis were used in this study. 
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These type of analyses have been used in similar studies dealing with the structure of trust instruments and 
identification of hypothesized variables (Sarker, Valacich & Sarker, 2003). 

Contexts and Participants 

Fifty-six undergraduate and graduate students participate in this study. They were enrolled in three different 
courses offered by a large U.S. Midwestern university and a renowned higher education institution in Denmark. For 
each course, students were selected randomly to the learning teams of three to five members each. Each team had to 
develop projects, which required the application of the knowledge and skills learned in the courses. The team 
projects were largely supported by different types of technology (e.g., learning management and videoconferencing 
systems). The undergraduate students were mostly from the USA apart from 3 students from Canada, Taiwan and 
South Korea. Their ages ranged 18 to 29, and they were mostly females. Graduate students showed a dispersed age 
range from 24 to 48, and distributed citizenship among USA, Turkey, India, Ukraine and Denmark. This sample was 
equally distributed between male and female. 

Data Collection Method 

Two versions of an online questionnaire on trust building in virtual learning teams were administered in 
two different times: at the mid-point of the project and after project completion. The questionnaires were created 
based on Mayer et al. (1995); and Aubert and Kesley’s (2003) instmments. Satisfaction with teamwork processes 
and products was used as a self-assessed measure of team performance. Questionnaire- 1 (34 items) was 
administered at the project mid-point and questionnaire-2 (37 items) after project completion. These two versions of 
the questionnaire had a demographic section. Items were answered in a scale of 1 to 7, with 1 being “strongly 
disagree” and 7 “strongly agree.” 

Reliability Analysis 

Cronbach’s alpha was computed to evaluate the reliability of the scores generated by the two versions of 
the questionnaire. As a result one item from the benevolence measure was dropped from the analysis. The alphas 
were all above .70 with the exception of the trust scale (.62) in questionnaire- 1. 

Data Analysis 

The data analysis consisted of two successive stages: (1) confirmatory factor analysis to assess the degree 
of fit between the data and the model, and (2) path analysis to understand the relationships between conceptual 
model key dimensions (ability, benevolence, integrity, satisfaction with team processes and products). 

Results and Conclusions 

Based on the confirmatory factor analysis, most of the observed variables measured the hypothesized 
constructs proposed in the theoretical framework. The results of the path analysis revealed that: 

(1) Ability and benevolence were strong antecedents of trast in virtual learning teams by the project mid- 
point. Surprisingly, integrity did not to have much impact at this point of time. 

(2) After project completion, benevolence followed by integrity were stronger antecedents of trust than 
ability. 

(3) As suggested by the theoretical framework, there was a clear relationship between trust building in 
virtual learning teams and outcomes, such as satisfaction with teamwork process and product. 

(4) Integrity had a stronger impact on satisfaction with the products followed by satisfaction with the 
processes. Benevolence showed a significant relationship with satisfaction with process but no 
relationship with satisfaction with product. 

As hypothesized by the theoretical model, all three antecedents of trust (benevolence, integrity and ability) 
had significant impact on trust building by the end of the team project. Ability and benevolence were the only 
critical elements identified at the mid-point of the project. Conversely to Aubert and Kesley’s (2003) explanation, 
good intentions can also build trust, as well as the ability to perform and deliver what is promised at any stage of the 
project. On the other hand, integrity did not seem to have a critical role in the formation of trust at the mid-point of 
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the project. Possibly, the perceptions of one’s firm adherence to a set of principles in learning teams are assumed 
when members attend the similar courses in equivalent programs and related institutions. 

Perceptions of benevolence or desire to do good towards one’s fellow team members had more impact on 
the final levels of trust than other antecedents of trust. This finding was consistent with what Schoorman, Mayer and 
Davis (2007) predicted; while perceptions of ability and integrity would develop relatively quickly, benevolence 
judgments would take more time. This is what happened when the virtual learning teams progressed in their project. 
Benevolence became the stronger antecedent of trust as the project moved towards the end. 

A number of previous studies found a significant relationship (e.g., Jarvenpaa, Knoll and Leidner, 1998 and 
Mayer et ak, 1995) between trust building in virtual teams and team performance, which is consistent with this 
study’s results. The results indicate that the higher the levels of trust, the higher the satisfaction with the project and 
the teamwork itself. 

Perceptions of integrity were related to team performance in terms of satisfaction with the final product. In 
a lower degree, a similar relationship occurred between integrity and satisfaction with team processes. Benevolence 
was also more noticeable in relation to satisfaction with team processes. Even though the team’s satisfaction with 
the quality of the outcomes was important, the satisfaction with the processes (e.g. professional growth and quality 
of interpersonal relationships) became the most important aspect of teams’ performance. This focus on the processes 
rather than on the products is not surprising in a learning team, where team members strive to gain knowledge and 
skills. In the same line of reasoning, ability did not have a significant relationship with the team performance 
because of the division of labor within a learning team. Its focus is on personal development, so members may take 
on tasks in their areas of weakness in order to develop new knowledge and skills. However, working teams focus on 
performance and products; thus, the members’ roles are closely associated with their greatest expertise. 
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ABSTRACT; 

This paper presents Dynamic Space, a software-based learning tool for creating and experiencing 
interactive concept maps. This tool allows learners with moderate interactive design skills to add advanced 
three-dimensional interactivity to two-dimensional concept maps. Dynamic Space is an open-source class 
set developed to enhance Adobe Flash®. On-going development, research, and implementation of Dynamic 
Space is based on the educational benefit of concept mapping as “a cognitive strategy particularly suited to 
identifying relationships between ideas” (De Simone, 2007, p.33). 

INTRODUCTION; 

Stimulating students to synthesize information into knowledge that can be applied is a key 
instructional challenge faced by educators in all disciplines. One method to encourage synthesis that is 
accepted across multiple disciplines is concept mapping. Concept mapping is “a cognitive strategy 
particularly suited to identifying relationships between ideas” (De Simone, 2007, p.33). 

A concept map is a graphic depiction of related knowledge (Wesley & Wesley, 1991) (figure 1). 
The relationship between knowledge ideas is typically hierarchical, constructed of information nodes and 
links (Jacobs-Lawson & Hershey, 2002). Links between nodes are often labeled with verbs to explain 
relationships (De Simone, 2007). 
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Concept Map 




figure 1: Example concept map (Rillero, 2006). 



A concept map can serve as a learning strategy, evaluative technique, and/or creative exercise. De 
Simone (2007) argues that the cognitive effort of externalizing information as a concept map works to 
synthesize knowledge into a fuller learning experience. Jacobs-Lawson and Hershey (2002) echo this idea 
by asserting that constructing concept maps requires students to understand content more fully that 
objective testing. Dansereau and Newbern (1997) advocate for concept mapping as means for personal 
expression. 

Two conventional approaches to creating concept maps include drawing maps by hand, or using 
dedicated concept mapping software, such as Inspiration® (De Simone, 2007). This paper presents 
Dynamic Space, a new software-based learning tool for creating and experiencing interactive concept 
maps. Unlike conventional approaches to concept mapping. Dynamic Space emphasizes interactivity as a 
means for enhancing learning outcomes. 

The remainder of this paper proceeds in the following manner. First, the authors discuss the 
theoretical framework from which Dynamic Space has been developed. Next, we outline its general 
functionality. Finally, the paper finishes with a brief discussion of implementation and future research. 

THEORETICAL FRAMEWORK: 

The pedagogical basis for Dynamic Space as a learning tool is derived from a constructivist 
approach to learning. This approach, which stresses the importance of “thinking in meaningful ways [...] 
such as representing what students know, rather than memorizing what teachers and technologies tell them 
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(Jonassen, 2000, p.24), is consistent with concept mapping. In the case of Dynamic Space, learning 
opportunities expand beyond the process of representing knowledge, and into the realm of interacting with 
knowledge. 

Access to these interactive learning opportunities is made possible by incorporating the 
philosophical tenets of phenomenology into the design rationale of Dynamic Space. Phenomenology is a 
subjective mode of inquiry addressing “the various layers of experience, and the different structures of 
meaning involved” (Berger & Luckmann, 1966, p.21). In this way. Dynamic Space allows learners to 
create unique structures of meaning through their own interactive experiences. 

These unique structures of meaning are products of Dynamic Space’s non-linear interactive 
environment. This environment, which requires investigation as a means for discovery, produces learning 
experiences that are unique onto each learner. Thus, Dynamic Space as a learning tool promotes 
individualized knowledge creation through personal experience. 

Integrated with this individualized approach to knowledge creation is research related to 
instructional gaming (Garris, Ahlers, and Driskell, 2002) and flow theory (Csikszentmihalyi, 1990). The 
influence of instructional gaming research on the development of Dynamic Space pertains to the benefits of 
learner engagement on learning outcomes (Garris, Ahlers, and Driskell, 2002). Their research investigates 
interactivity as related to games, motivation, and learning. “To the extent that we pair game features with 
appropriate instructional content and practice, we can harness these motivational forces to achieve desired 
learning outcomes” (Garris et ah, 2002). Additional support for game-like features as means to enhance 
learners’ experiences is found in flow theory. “The more [an activity] inherently resembles a game - with 
variety, appropriate and flexible challenges, clear goals, and immediate feedback - the more enjoyable it 
will be regardless of the [person’s] level of development” (Csikszentmihalyi, 1990, p.l52). Thus, by 
incorporating game-like keyboard-based controls, animation and a puzzle-like structure, learners are 
encouraged to relate to an interactive concept map as if it were a game. The result of this approach is a 
learning tool with expanded learning opportunities, designed to be enjoyable for the learner. 

FUNCTIONALITY; 

Dynamic Space is an open-source class set developed by the authors to enhance Adobe Flash®. 
Dynamic Space allows learners with moderate interactive design skills to add advanced three-dimensional 
interactivity to two-dimensional concept maps. Dynamic Space offers two different modes of interaction: 

1) the designer of an interactive concept map, 2) a user of an interactive concept map. Learning 
opportunities exist for both modes. 

Designers begin by creating two-dimensional vector-based concept maps either directly in Flash, 
or in third-party programs such as Adobe Illustrator®. If a third-party program is used, designers must 
import their concept maps into Dynamic Space, using Flash. Designers then add interactivity to their 
concept maps using the Dynamic Space class set within the Flash authoring environment. Finally, designers 
publish their concept maps for members of the user audience.* 

The primary function of Dynamic Space is as a template for establishing interactive relationships 
between elements of a concept map. The Dynamic Space download package uses simple logic and basic 
programming to allow designers to create animations that reinforce the relationships between elements. For 
example, if a user selected a particular element of a concept map, that action can be designated to initiate 
the visibility of related elements. This added layer of experience allows both designers and users to 
construct new meaning (Berger & Luckmann, 1966). 

The secondary function of Dynamic Space is as a scrolling and zooming environment that displays 
a concept map. Unlike many three-dimensional interfaces. Dynamic Space uses scale to add a third 
dimension (figures 2 & 3). The more conventional representation of a third dimension uses depth. 



‘ An example concept map and additional instructions are included in the download package: 
https://netfiles.umn.edu/users/frah0005/www/dvnamicSpace/dvnamicSpace.html 
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figures 2 & 3: Scrolling and zooming. 
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Users of a Dynamic Space-enhanced concept map are able to scroll around and zoom in on 
sections of the map. In the context of learning, using scale to create three-dimensional interaction provides 
an advantage over using depth. This advantage is due to avoiding the excessive cognitive load of depth’s 
visual metaphors, include deciphering parallax and understanding modes of locomotion. In this way. 
Dynamic Space functions under the relatively simple principle of resolution; a user’s sense of perspective 
and location remain the same, what changes is the scope. As a result, a user’s cognitive resources are not 
diverted. 

Adding interactivity to conventional concept mapping is beneficial with respect to engagement for 
both the user and the designer. Interactivity requires engagement through user investigation (Andersen, 
2004), which is linked to the higher-order thinking skills of analysis and evaluation (Bloom, 1956; 
Anderson & Krathwohl, 2001). Interactivity also allows for a new level of creativity for the designer. 
Creativity is a form of synthesis (Bloom, 1956), and represents the highest-order thinking skill (Anderson 
& Krathwohl, 2001). 
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IMPLEMENTATION AND EUTURE RESEARCH: 

A key opportunity for implementing Dynamic Space is in the context of computer aided visual 
communication (i.e., art, design, architecture). Students of these disciplines regularly engage with 
principles integral to creating compelling concept maps (i.e., visual hierarchy, typography, balance 
relationships). Many of these students possess moderate interactive design skills. 

In addition to these disciplines, regular use of interactive technology continues to expand into 
more areas of education. Eor example, at the University of Minnesota, the School of Journalism and Mass 
Communication also plays host to the Institute of New Media Studies (INMS), which focuses on 
“innovative forms [of communication and] the impact of these changes in the media landscape” (INMS, 
2008). 

In light of this expansion. Dynamic Space’s template -based approach to interactivity is designed 
to allow for a low barrier to entry for students from various disciplines. This is particularly true with respect 
to the advanced level of interactivity inherent to Dynamic Space. It is a learning tool designed to reward 
students, regardless of discipline, by providing a maximum return on their invested effort. 

Euture research opportunities for Dynamic Space exist in two main areas. First, additional 
development of the open-source architecture could address both performance increases and additional 
features. Potential performance increases include the optimization of rendering functions and animation. 
Additional features could include database integration. 

Second, preliminary investigation indicates strong potential for Dynamic Space to be used in a 
qualitative study of student design practices. Initial discussion are underway to allow college graphic 
design majors to use Dynamic Space in the context of creativity training. But considerable development 
must be done on the Dynamic Space architecture before this can become feasible. 

SUMMARY: 

This paper presents Dynamic Space, a software-based learning tool for creating and experiencing 
interactive concept maps. Concept mapping is a strategy that is accepted across multiple disciplines as a 
means for creating knowledge. As described above, the addition of interactivity to conventional concept 
mapping allows for expanded learning opportunities for both designers and users. 

A key benefit of interactivity is the increased engagement with which learners meet these 
opportunities. With Dynamic Space, increased engagement is achieved by developing the interactive 
experience to possess game-like features. These features have been shown to lead to increased learner 
motivation and enjoyment. 

Dynamic Space is a learning tool designed to accommodate learners with a wide range of 
technical skills. The simple logic and basic programming contained in the Dynamic Space download 
package is oriented toward learners with moderate interactive skills. The open-source structure of this 
architecture is intended to promote augmentation and expansion of its functionality by learners with 
advanced programming skills. 

REFERENCES: 

Andersen, K. (2004). It Felt Like Clown Sparkles. Interactions, Vol. 11(5), pgs. 61-63. 

Anderson, L. W., Krathwohl, D. R. (2001). A taxonomy for learning, teaching, and assessing: A revision of 
Bloom’s taxonomy of educational objectives. New York: Longman. 

Berger, P., Luckmann, T. (1966). The social construction of reality. New York: Doubleday. 

Bloom, B. (1956). Taxonomy of educational objectives. New York: Addison Wesley. 

Csikszentmihalyi, M. (1990). Flow. New York: Harper Perennial. 

Dansereau, D., Newbern, D. (1997). Using Knowledge Maps to Enhance Teaching. New Paradigms for 
College Teaching. Edina: Interaction Book Company. 

De Simone, C. (2007). Applications of Concept Mapping. College Teaching, 55(1), p. 33-36. 



254 




Garris, R., Ahlers, R., Driskell, J. (2002). Games, motivation, and learning; A research and practice model. 
Simulation & Gaming, 33(4), p.441-467. 

Institute of New Media Studies (2008). About the Institute for New Media Studies. Retrieved 3.10.08 from 
http;//www. inms. umn.edu/pages/view. asp?id=l 1. 

Jacobs-Lawson, J., Hershey, D. (2002). Concept Maps As an Assessment Tool in Psychology Courses. 
Teaching of Psychology, 29(1), p. 25-29. 

Jonassen, D. (2000). Transforming Learning with Technology: Beyond Modernism and Post-Modernism or 
Whoever Controls the Technology Creates the Reality. Educational Technology, 40(2), p.21-25. 

Rillero, P. (2006). Concept map. Retrieved 3.10.08 from http://www.west. asu.edu/rillero/. 

Wesley, W., Wesley, B. (1991). Concept mapping: A Brief Introduction. Faculty Development. 



255 




Applying a Cognitive Apprenticeship Approach to Developing the Technology 

Self-efficacy of Pre-service Teachers 

Mark Jones 

Oklahoma State University 
102C Willard Hall 
Stillwater, OK 74078 

Pasha Antonenko 
Oklahoma State University 
210 Willard Hall 
Stillwater, OK 74078 

Introduction 

It is widely acknowledged today that a student’s comprehensive education must include technology literacy. 
Recently, ETS piloted their ICT (Information and Computer Technology) Literacy Assessment. The results indicate that 
college students and high school seniors are far from technology literate (ETS, 2006). 

Defining technology literacy has proven to be challenging in the past, however development of national technology 
literacy standards have helped to be able to identify what it means to be technology literate. The National Education 
Technology Standards for Students (NETS-S) developed by the International Society for Technology in Education (ISTE) is 
a leading example of widely recognized technology literacy standards (ISTE, 2007). The NETS-S identify six areas of 
technology use that students should develop competence, including I) basic operations and concepts, 2) social, ethical, and 
human issues, 3) technology productivity tools, 4) technology communication tools, 5) technology research tools, and 6) 
technology problem-solving and decision-making tools (ISTE, 2007). In June of 2007, ISTE is releasing an updated version 
of the NETS-S standards, introducing a new standard focusing on using technology for creativity and innovation (Borja, 
2007). 

Enabling students to achieve technology literacy as defined by the NETS-S requires that they be exposed to learning 
environments that foster the development of technology skills. ISTE proclaims that the conditions necessary for creating this 
environment hinge on technology leadership of teachers and administrators (ISTE, 2007). Accommodating these conditions 
clearly involves a systemic approach to creating learning environments that are conducive of technology literacy. There is no 
question of the public’s attitude towards teaching technology literacy to students. A recent Gallup Poll indicated that most 
Americans feel that technology literacy is very important and should be taught in school (Russell, 2005). Unfortunately the 
will of the public does not always translate to recognized responsibility of schools. 

It seems that most K-I2 teachers do not feel an individual responsibility for the technology literacy of their students. 
This responsibility is usually passed along to someone else, such as a computer teacher. This mentality conflicts with the 
necessity for students to see technology in use and to use technology throughout their classes in order to develop true 
technology literacy. The solution lies in teacher education programs preparing new teachers to possess the skills and 
dispositions needed to effectively integrate technology in the classroom. Unfortunately when it comes to technology in 
teacher education, as Lock (2007) states, “ICT has been delivered as stand-alone introductory courses, has been marginalized 
within programs, and suffers from an absence of modeling” (p. 2). 

In Eall 2003, 100 Preparing Tomorrow’s Teachers to Use Technology (PT3) leaders were invited to a summit 
intended to establish a research agenda for identifying “best practices” in preparing teachers to effectively integrate 
technology (Thompson, 2005). It is the intention of this article to show that current literature provides strong evidence that 
these best practices are those that are encompassed by the cognitive apprenticeship model. 

Cognitive Apprenticeship 

Cognitive apprenticeship is an instructional model that is derived from many aspects of sociocultural learning theory 
(Collins, Brown, & Holum, 1991; Paz Dennen, 2004). This model is particularly applicable to learning complex skills, such 
as how to effectively integration technology into teaching and learning. Three key characteristics of cognitive apprenticeship 
include making thinking visible, situating learning in an authentic context, and encouraging generalization and transfer 
through providing a range of experiences and through reflection and articulation (Ghefaili, 2003). 
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When learning to be a chef, an apprentice works in a kitchen performing the tasks of a chef. These tasks begin very 
basic, and gradually the apprentice is given more responsibility for more difficult tasks. While developing these skills, the 
master chef models how to do the skills, and then coaches the apprentice in performing the skills. Through this process of 
modeling, scaffolding, and fading, the apprentice develops the ability to perform the skills on their own. In addition, the chef 
apprentice rarely works in isolation, but instead is learning the skills of the trade alongside other apprentices in a community 
of practice. The cognitive apprenticeship model ports the attributes of traditional apprenticeship into learning complex 
cognitive skills. 

In the design of learning environments based on the cognitive apprenticeship model, the primary instructional 
strategies employed are modeling, scaffolding, coaching, articulation, reflection, and exploration. In addition, the social 
context of learning is strongly recognized and exploited in the apprenticeship approach (Collins et ak, 1991). 

The cognitive apprenticeship model has been applied to many areas of learning, such as reading, writing, and math. 
In review of literature on how teacher education programs are attempting to prepare technology literate teachers, a clear trend 
emerged that offers solid evidence for the benefits of a cognitive apprenticeship approach. The remainder of this article will 
present examples of this research, and discuss the implications for how teacher education programs can improve their efforts 
to develop technology literacy in pre-service teachers. 



Situated Learning 

In an apprenticeship, the learner is actively engaged in tasks that directly relate to the skills they are learning. Lave 
and Wenger (1991) refer to this as situated learning, and how it lends to legitimate peripheral participation. Legitimate 
peripheral participation describes how a novice develops experience within a community of practice. Tasks and 
responsibilities begin very basic, and gradually the apprentice is given more responsibility for more difficult tasks. 

Key points of situated learning include embedding knowledge into a relevant context, including authenticity or real- 
world application, making learners active in the learning (Wertsch, 1991), developing communities of practice (Rogoff, 

1990), and recognizing shared or distributed cognition (Ghefaili, 2003). 

In terms of pre-service teachers learning technology, situated learning theory explains the importance of learning 
technology skills in tandem with learning how to integrate technology into instruction. When technology skills alone are the 
focus, this lends to decontextualized technology literacy. 

According to Finn (1999), focusing on basic literacy skills only creates a domesticating form of literacy, in contrast 
with empowering literacy skills, those which prepare a person to critically analyze information and synthesize new 
information. Applying this to technology literacy would mean that focusing on basic, decontextualized technology skills 
would lend to domesticating technology literacy. When translated into long-term technology behaviors, teachers with this 
level of literacy will likely only use what technologies they were taught and would struggle to integrate technology 
effectively. This is supported by literature discussing how a narrowly perceived view of educational technology limits the 
perceived value of educational technology (Okojie, Olinzock, & Okojie-Boulder, 2006). 

In regards to technology, many teacher education programs focus on the mastery of hardware and software. This 
often results in students who acquire technology skills, yet do not feel confident in their ability to teach in a technology-rich 
environment (Brown & Warschauer, 2006). As Jacobsen and Lock (Jacobsen & Lock, 2004) state, “It is clear that training 
teachers how to navigate computer applications is insufficient for meaningful technology integration”. 

A study by Banister & Vannatta (2006) supported placing the burden on the student to enter the professional 
education program with a prerequisite level of technology skills, and using a technology skills assessment as a gateway to 
entrance into the program. In this way, the technology training within the program can focus primarily on applying 
technology to a pedagogical purpose. While this approach may be convenient, according to sociocultural learning theory, 
skills learned in a computer applications course are often taught as decontextualized procedures. This would result in 
difficulty of transferring these skills to pedagogical technology integration. 

Modeling 

Bandura’s Social Learning Theory explains the value of vicarious learning, and the importance of modeling in 
learning. According to Bandura, learning from modeling occurs through the processes of attention, retention, motivation, and 
finally reproduction of what was being modeled (Driscoll, 2000). 

Many studies have validated the importance of modeling of technology integration to pre-service teachers (Barnett, 
2006; Basham, Palla, & Pianfetti, 2005; Dexter, Doering, & Riedel, 2006; Rowley, Dysard, & Arnold, 2005; Swain, 2006). 
The closer a learner identifies with a model and the perceived level of competence of the model both have an affect on the 
effectiveness of the model on the learner (Driscoll, 2000). In pre-service teacher education, the professor of the content area 
methods courses would have a very strong modeling influence on the pre-service teachers. The pre-service teacher 
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recognizes that very soon be in front of students as a teacher, so the knowledge and skills of the methods professor has high 
value, resulting in a high degree of attention, retention, and motivation of the pre-service teacher. Because of this, it is 
imperative to the development of educational technology skills and dispositions in pre-service teachers that professors in the 
content-area methods courses model effective technology integration. 

A key aspect of modeling in a cognitive apprenticeship is making visible the thinking of the expert to the novice 
(Collins et ah, 1991). For example, Swain (2006) describes the benefits of thinking aloud with pre-service teachers about 
how to integrate technology. Doing this facilitates education students to begin to think more like education producers than 
education consumers, and to see themselves in the role of the teacher action researcher (p. 56). 

Modeling how to effectively integrate technology can also include video-based models (Rowley et ah, 2005). A 
study at the University of Dayton integrated the use of technology modeling through PBS’s TeacherLine professional 
development program. The level of effectiveness of the video-based modeling was dependent on how faculty implemented 
them, correlating positively with frequency of use of the videos. The study concluded that video-based modeling can serve 
as an effective way to demonstrate technology integrate to pre-service teachers and faculty, and an indirect benefit is the 
modeling of technology inherent in the implementation of the videos into instruction. Barnett (2006) observed similar results 
of pre-service teachers highly valuing video-based modeling of teacher practice. 

Being on the doorstop of entering professional practice, it is common for pre-service teachers to place high value on 
the pedagogical beliefs of the content area methods course. Non-behavior can serve as a form of modeling as well as overt 
modeling. Swain (2006) observed that while pre-service teachers obtained technology skills after participating in an 
introductory technology course, surveys indicated that many of them did not view integrating technology as worthwhile. 
Swain proposed that a possible explanation of this was a result of transmission of the beliefs and attitudes of teacher 
education faculty towards technology. 

Being able to model technology use places a burden of responsibility on teacher educators to acquire technology 
skills. Even if faculty have technology skills, they may not use these skills in their teaching (Aust, Newberry, O'Brien, & 
Thomas, 2005). Faculty face many of the same challenges of pre-service teachers to learning effective technology integration 
strategies. Some professors have even been willing to enroll in an educational technology course to be better prepared to 
model technology (Bird & Rosaen, 2005). 



Coaching 

To explain the role of social interaction between a novice learner and a more capable other, Vygotsky proposed the 
concept of the zone of proximal development (Rogoff, 1990). In this relationship, the expert supports the novice, enabling 
the novice to accomplish more than is possible on their own. While in the zone of proximal development, the novice is 
challenged just beyond their independent abilities, but not so much that they feel hopeless. 

The support given to the learner is referred to as scaffolding (Rogoff, 1990). Just as traditional scaffolds provide 
support for a worker to reach higher than they could on their own, cognitive scaffolds support learners as they learn new 
knowledge and skills. Scaffolds can take many forms, such as bowling alley bumpers, bicycle training-wheels, or partially 
worked math problems. 

In the cognitive apprenticeship model, the task of the expert continually monitoring, providing feedback, and 
adjusting difficulty is referred to as coaching (Ghefaili, 2003). The coaching process involves a periods of scaffolding and 
fading of support to a point where the learner is successful in performing the skills independent of the instructor’s support 
(Collins et ah, 1991). 

In learning to use technology, the role of coaching is especially important. Barton & Haydn (2006) noted in their 
study that, “If no one is able to show the trainee confidently how to assemble the ICT resources and then to provide examples 
of how to exploit these resources within the classroom, this places a huge burden on the inexperienced trainee” (p. 266). In 
addition to all the other aspects of teaching that a new teacher is coming to terms with, it is no surprise that without a role 
model’s support, many pre-service teachers are unwilling to commit to “go it alone” in developing the skills to effectively 
implement technology. 

Lock (2007) found that onsite, online, and ongoing support of pre-service teachers engaged in inquiry-based digital 
projects. Having a combination of one-to-one, one-to-small group and one-to-large group provided the necessary level of 
responsive support to the novice technology users. Lock’s recommendation for adequate support of educational technology 
learning also requires that the professor recognize when it is necessary to enlist additional support. This can be a challenge 
for many professors who fear the thought of appearing incompetent in any way to their students. 
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Community of Practice 



The sociocultural viewpoint of attaining technology literacy is that developing the skills and dispositions of a 
technology literate teacher must occur collaboratively “in specialized user cultures and communities” (Talja, 2005, p. 20). A 
key component of the cognitive apprenticeship model is articulation (Collins et ak, 1991). The act of a learner articulating 
their understanding of the skills to be learned is inherently a social process. As Bender (2003) states, “Collaboration is vital to 
learning so that students understand questions, develop arguments, and share meaning and conclusions among a community 
of learners” (p. 8). Through dialogue with other learners and the instructor in a community of practice, learners communally 
build on each others’ understandings through distributed cognition (Ghefaili, 2003). “Learning becomes a process of 
reflecting, interpreting, and negotiating meaning among the participants of a community. Learning is the sharing of the 
narratives produced by a group of learners” (Ghefaili, 2003, p. 7). 

The role of collaboration in the cognitive apprenticeship model is framed by Lave & Wenger’s (1991) concept of 
legitimate peripheral participation. This concept describes the process of membership in a community of practice (Driscoll, 
2000). In a community of practice focused on learning educational technology, an inexperienced pre-service teacher would 
be considered a newcomer, while the educational technology faculty would be considered oldtimers, those who have full 
access to the resources of the community. Other education faculty would likely fall somewhere in between, preferably more 
towards an oldtimer than a newcomer with educational technology. Through participation in this community of practice, a 
pre-service teacher gradually grows in technology skill and self-efficacy, from a newcomer towards full membership. 

Many studies have supported the involvement of pre-service teachers in communities of practice specific to learning 
to integrate technology. In the study mentioned previously. Lock commented on the fact that throughout their inquiry-based 
project development, the pre-service teachers worked collaboratively to negotiate their understanding and use of technology 
in both their project and for presenting their co-created knowledge (Lock, 2007). She asserts that this collaboration engaged 
the learners in higher order thinking and problem solving in learning about technology. Working in small groups engaged the 
learners in “rich dialogue, thoughtful reflection, and critical thinking. According to Rogoff (1990), this collaboration requires 
that the learners maintain a level of intersubjectivity to be able to effectively communicate their developing understanding of 
technology among each other and their instructor. This task in and of itself serves a metacognitive role in the pre-service 
teacher’s self-assessment of their technology skills. 

Bazeli (as cited in Okojie et ak, 2006) states that when learners are involved planning and implementing learning 
about technology “the burden is lifted from the teachers and the learning process becomes collaborative, with the teachers 
assuming the role of facilitator rather than a disseminator of information. Further, as students are actively involved in 
planning and implementing technology production, they gain critical thinking and problem-solving skills along with 
curricular learning” (p. 4). This coincides with Lock’s comments towards collaboratively learning technology. 

In their study. Barton & Haydn (2006) observed that having pre-service teachers collaboratively learn educational 
technology was a “time-effective, high-challenge low-threat learning environment for trainees” (p. 265). They noted that 
group work enhanced the trainees’ ICT planning skills, but more importantly, group work provided moments of enthusiasm 
in using ICT. The social nature inherent in learning explains why learning technology collaboratively would produce these 
results. 

Communities of practice focused on educational technology can also be between teacher education programs and K- 
12 schools, as was demonstrated by the Teacher Inquiry Group (TIG) (Murphy, Richards, Lewis, & Carman, 2005). This 
group worked collaboratively to determine best practices for preparing pre-service teachers to integrate technology. Through 
these kind of partnerships, teacher educators can be more in tune with the locally valued technology skills that are sought 
within the school culture that most of the pre-service teachers will soon find themselves employed. 

Attitude and Self-efficacy 

Self-efficacy is a component of Bandura’s social learning theory (Driscoll, 2000). It describes a level of confidence 
one has in his or herself to be able to do something. Studies have shown that a major factor that determines if teachers 
implement technology is their attitude toward the benefits of technology, as well as their self-efficacy in using technology. In 
relation to the evidence already presented for learning technology skills in a meaningful context, research has shown that 
“perceived increases in technological skill do not change the degree to which an individual values the integration of 
technology in the classroom” (Basham et ak, 2005, p. 274). In other words, even if pre-service teachers acquire technology 
skills, if they do not value the use of technology to improve teaching and learning, those technology skills will not translate 
into classroom integration. 

Wozney, Venkatesh, and Abrami (2006) examined the motivational factors that impact whether teachers will 
implement technology in their classroom. Their study determined that the strongest factors in predicting levels of technology 
integration were expectancy of success and perceived value. They also note that technology implementation is a “dynamic 
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process mediated by subjective teacher characteristics and by conditions within the school” (p. 192). The best-designed 
instruction can only do so much to convince teachers to be technology implementers. 

Talja (2005) states, . .in different contexts of discussion, speakers adopt different perspectives on IT 
competencies” (p. 20). This points out the social and discursive nature of attitudes towards technology skills, and explains 
why in many cases, even if technology skills are present, they may not be implemented into classroom practice. Engaging 
pre-service teachers in learning technology that directly applies to their content area has shown to increase technology self- 
efficacy and improve attitudes towards future use of the technology (Gado, Ferguson, & van 't Hooft, 2006). 

Kadijevich (2006) studied the level of pre-service teachers’ interest in attaining educational technology standards 
based on their attitude towards technology and the support they received in attaining the standards. It was found that interest 
was mediated by attitude, and the effects of support on attitude were positive and significant. Besides demonstrating the role 
of attitude in technology literacy, this data adds evidence of the importance of scaffolding pre-service teachers in attaining 
these skills. 

Frequency of computer use has also shown to be a significant predictor of technology self-efficacy (Chu, 2003; 

Pope, Hare, & Howard, 2005). The findings of Chu’s study “confirmed the notion that increasing the opportunities to learn 
and use a computer may facilitate the confidence and competence of pre-service teachers” (p. 140). That study reinforced the 
importance of providing pre-service teachers with many experiences with technology throughout their teacher education 
program. As will be shown in the next section, isolating teacher technology training to a single educational technology 
course is not sufficient to provide pre-service teachers with the amount and variation of experiences needed to develop strong 
educational technology self-efficacy. 



Range of Experiences 

A key question to consider in pre-service teacher technology training is how to coordinate the exposure of pre- 
service teachers to technology training. Many programs use a single course model. By this method, there is a specific course 
designed to train pre-service teachers in educational technology. This concentrated approach provides very limited time and 
space for pre-service teachers to develop skills and dispositions necessary for transferring to actual classroom implementation 
(Pope et ah, 2005). According to Jacobsen & Lock (2004), “One-size -fits-all workshops and stand-alone courses are 
limiting and tend to fragment professional development with little or no ongoing support” (p. 78). As shown by what has 
been discussed already in consideration of the cognitive apprenticeship approach, many problems are inherent in a single 
course being the sole source of technology preparation for pre-service teachers. 

Primarily, there is simply not enough time in a single course to adequately prepare pre-service teachers to acquire a 
level of skills and confidence for them to effectively integrate technology into their classroom. Development of the complex 
skills involved in effectively integrating technology takes a tremendous amount of time and practice. When developing ICT 
literacy is forced into a short timeframe, negative consequences can result. Barton & Haydn (2006) note that there is a danger 
of information overload that pre-service teachers can experience from trying to learn too much ICT in too short of a time, a 
consequence of which causes pre-service teachers to feel very unsure about their ability to use any technology at all in their 
classroom. 

Research suggests that the most effective approach is to systemically integrate technology throughout the teacher 
education program. As stated by Jacobsen & Lock (2004), “Sustainable technology integration over the long-term requires 
cross-disciplinary modeling of effective technology integration across all subject area within a teacher education program” 

(p. 95). Correlating directly with the cognitive apprenticeship model, this approach provides the depth and breadth of 
experiences that pre-service teachers need to build educational technology self-efficacy. 

The Ed-U-Tech project (Dexter et ak, 2006) employed a dual integration approach. Technology was integrated into 
the content-specific methods courses as well as the content-areas being integrated into the educational technology course. By 
doing this, pre-service teachers receive a range of experiences with educational technology and how it can be used in their 
specific content area. 

In addition to dual integration, the Learning Generation model (Aust et ak, 2005) used a cohort-based approach. 
Cohorts consisted of faculty from teacher education as well as general studies, pre-service teachers, in-service teachers, and 
K-12 students. The cohorts developed through seven stages: genesis, consultation, planning, initiation, action, assessment, 
and celebration. This approach positions education technology faculty in the position of expert consultants, advising the 
cohorts as to what the possibilities are and guiding the cohort as they implement a collaboratively planned technology 
integrate project. The cohort then implements and assesses their educational technology product. 

Survey results showed that this approach did increase the participants’ confidence in their ability to use technology 
in instruction. A gap was noticed between men and women, with men having a higher increase in confidence. To address 
this, recruiting of female faculty for cohorts increased, as well as increasing the number of female faculty who are modeling 
effective use of technology (Aust et ak, 2005, p. 186). 
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As noted in their study, a model such as this would require a tremendous amount of support at many levels. 
Recognizing this, the authors state, “Sustainable reform must involve systemic conditions where representatives from the 
diverse stakeholders are actively involved in developing and owning the innovation” (Aust et ah, 2005, p. 171). 

Other studies support the extension of technology training into the pre-service teacher’s field experiences. Brown & 
Warschauer (2006) found that the role of mentor teachers was pivotal to the developing attitudes of pre-service teachers 
towards technology. They state, “the use of educational technology by the pre-service teachers is strongly associated with 
observing proficient mentor teachers who model technology-enriched instruction” (Brown & Warschauer, 2006, p. 617). 

The idea of placing pre-service teachers only with mentor teachers who effectively integrate technology would be 
challenging at best. In recognition of this. Brown & Warschauer (2006) note that even collaboration by asynchronous 
electronic communication can provide a level of support to pre-service teachers as they attempt to implement technology in 
their field experience. 



Recommendations 

The research presented in this article has provided evidence in support of a cognitive apprenticeship approach to 
technology integration in pre-service teacher education. Other reviews of literature have observed similar trends in what 
strategies have proven effective. In a literature review of 68 studies on methods used to incorporate technology into pre- 
service education, Kay (2006) found ten key strategies, including “delivering a single technology course, offering mini- 
workshops, integrating technology in all courses, modeling how to use technology, using multimedia, collaboration among 
pre-service teachers, mentor teachers and faculty, practicing technology in the field, focusing on education faculty, focusing 
on mentor teachers, and improving access to software, hardware, and/or support” (p. 383). The most important fact to note is 
that Kay’s study revealed that programs which implemented four or more of these strategies experienced a much greater 
effect on pre-service teachers’ computer use. Kay also noted that most of the research on this topic had severe limitations, 
either from research design, poor instruments, and poor samples. Clearly, this shows that much more rigorous research 
remains to be done in this area. 

Oklahoma State University’s College of Education is piloting a program that intends to apply the cognitive 
apprenticeship approach to developing pre-service teachers’ efficacy of teaching with technology. The goal of this program 
is to prepare teachers to effectively implement student response system technology (clickers) to increase the engagement of 
learners during instruction. Clickers are a particularly effective technology to base this program on due to the fact that using 
clickers incorporates many important aspects of good pedagogy, such as engagement, questioning, collaboration, feedback, 
and assessment. 

This program implements a cognitive apprenticeship approach by allowing students to begin at the periphery of a 
community of practice and gradually move inwards as their skills and confidence in teaching with clickers builds to the point 
that they can effectively teach with the tool as well as create or adapt lessons that are augmented by the tool. The ultimate 
expected outcome of the program is that pre-service teachers’ experiences with clicker-augmented instruction will be 
transferred into their own classroom practice. As discussed previously, the translation of technology skills to classroom 
practice is something that has been difficult to accomplish through the single educational technology course approach. 

This program is initiated by first modeling effective use of clickers during teaching methods courses. During this 
time, the students see clickers used to effectively improve instruction in a situation near to that which they themselves will be 
using the tool. The students have many opportunities to see how the tool works to improve the instruction of the subject they 
are most interested in. Modeling of clickers in pre-service teachers’ subject area is crucial to the desired outcome of pre- 
service teachers who want to try the technology in their own teaching. 

Next, students enter the coaching phase of the program. Students are given focused training on the technical aspects 
of using the tool and its associated software. Students are given the opportunity to build lessons that include a series of 
clicker-based questions, and they also practice using these lessons among their peers, gaining feedback for improvement of 
the lessons and for building confidence during the process. 

Then, after the modeling and coaching phases, students take the clickers and their lessons into a real classroom 
setting to implement in an authentic setting. During this time they continue to receive feedback from the cooperating teacher 
as well as the attending university supervisor. This is the scaffolding and fading stage. By this point, students have had 
several opportunities to become comfortable and confident in using the tool during instruction. 

By design, the students develop their skills and confidence in teaching with clickers in collaboration and cooperation 
with peers in a community of practice, as opposed to trying to do this in isolation from the feedback and support gained 
through the community environment. This avoids the high-stakes “sink or swim” mode of attempting to teach with 
technology for the first time, especially for those at the beginning stages of teaching at all. As a result, we expect to see pre- 
service teachers, as well as their students, to have a much higher probability of experiencing instruction that is effectively 
enhanced through the use of technology. 
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Conclusion 



This article has discussed how a cognitive apprenticeship model is very effective for preparing technology literate 
teachers. It must be remembered that the importance of technology literacy in pre-service teacher education is that these 
teachers will eventually design technology-rich learning environments that will facilitate technology literacy of K-12 
students, such as those defined by ISTE’s NETS-S (ISTE, 2007). 

While the focus of this article is on pre-service teacher preparation, the implications of recommending a cognitive 
apprenticeship approach create a prerequisite to the modeling and coaching of technology skills necessary for this to work. 
The prerequisite is that teacher education faculty must also be willing to learn how to effectively integrate technology into 
their own teacher education courses. 

Many barriers have been identified that hinder faculty in developing technology skills and effectively integrate them 
into teaching. These barriers can be summarized as being related to problems of time, support, models, infrastructure, and 
culture/tradition (Brzycki & Dudt, 2005). 

To accommodate and overcome these barriers, professional development for faculty and in-service teachers must 
also consider the benefits of a cognitive apprenticeship approach. Wells (2007) states, “Eor educational organizations 
addressing instructional technology innovations, the traditional PD format is not conducive to teachers’ integration and 
adoption of new knowledge and skills presented” (p. 103). Most often occurring in the form of single topic workshops or in- 
services, this form of professional development fails to achieve any long-lasting affect on teachers to integrate technology. 

Eor all the reasons cited previously, this form of training fails for lack of context, coaching, and collaboration. What 
modeling that does occur fails to affect attitude or self-efficacy due to lack of relating to the trainer or the tool being taught is 
not presented with a clear pedagogical use in mind. 

Just as the cognitive apprenticeship model informs how to best prepare pre-service teachers to effectively integrate 
technology, it is also applicable to the design of professional development for teacher education faculty to be able to perform 
the modeling of technology that is such a crucial component of preparing technology literate teachers. 

Professional development in the use of technology for education faculty is often provided by an external department 
or by the college’s technology support staff. Each option presents a solution that is not sufficient to develop the level of 
technology literacy needed for education faculty to effective models of technology use in the classroom. External 
departments usually focus on a workshop model, while the technology support staff is usually only answering technical 
questions or troubleshooting hardware or software issues. 

There is a solution to this dilemma. Most colleges of education have one or more faculty who specialize in 
educational technology. When educational technology faculty enter the educational technology community of practice within 
their college, then these “oldtimers” provide the best option for successfully facilitating other education faculty to effectively 
integrate technology. Eollowing the cognitive apprenticeship approach, the educational technology faculty can develop long- 
term mentoring relationships with other education faculty that will yield meaningful acquisition of technology skills and 
development of positive attitudes toward the implementation of technology into instruction. 

This article has demonstrated that the cognitive apprenticeship approach holds great potential for developing 
technology literacy throughout the mentor-apprentice chain. Educational technology faculty model and coach technology use 
to other education faculty, education faculty model and coach technology use to pre-service teachers, and finally, these pre- 
service teachers eventually model and coach technology use to K-12 students. The end result accomplishes the initial goal, 
developing technology literacy of students to be successful in a technology-rich society. 
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The purpose of this paper is to generate a thorough understanding and consideration regarding the 
concept of presence in learning and its relevant studies. To be more specific, the first part presents readers with 
a general concept of presence, followed by a narrowed interpretation in the research field of educational 
psychology. The second part explores deeper into the models of presence in learning, referenced by a 
comparison of two presence models in learning. In the third part, evidence and research methods from previous 
empirical studies have been reviewed. Finally, the authors conclude the discussion with some implications and 
suggestions for further studies. 



The Concept of Presence 

In any instructional situation, learners are surrounded by a certain atmosphere which leads them to 
perceive the environment as if they were in a place where some other agents, most likely the teacher, classmates 
or other environmental forces, may as well have influenced to create. The basic assumption is that these 
learners are best educated when fully absorbed in the educational environment, which then leads them to 
experiencing the surroundings (Kolb, Boyatzis, & Mainemelis, 2001; Urdan & Midgley, 2001). As the 
boundary of educational scenes had extended from simple physical classroom to the technology-driven virtual 
space, the main focus in this field of study has been about how to create a collaborative, supportive environment 
(Stacey, 1999). Moreover, as people become more accustomed to using the Internet and starts to live a second 
life in the world it has created, it is not surprising to find out that in this new world a place for education should 
exist. 

From this new educational context arose the curiosity of presence in that whether learners are indeed 
engaged in learning (Wang & Kang, 2006). The concept is not entirely new. In fact, evidences were found in 
literature that the concept of presence existed before the technological improvement. For instance, 
philosophical interest in the post modern period may have preceded the technological change. The essence of 
this philosophical change originated from doubts that have been made about one's own existence. This change 
led people to have the constructive, or deconstructive, views on the world, which then gave them a probable 
ground for thinking that the ‘another’ world can be created with their own hands. This notion fundamentally 
supports the importance of presence since the value of this second world depends on how much it resembles, 
recreates, or enhances what is present. Furthermore, the concept of presence has been spotlighted in the field of 
computer science as in, for example, the second identity (i.e.. Avatar) of the virtual world. 

Seemingly, a variety of interpretations are possible among researchers of the relevant areas. However, 
the core of these various accounts of presence is one’s sense of ‘being there’. Then consequently, a question of 
“being where?” arises. Presence can be found in any place where the experiencer’s perception of the world 
differs from the actual world. Therefore, to understand the term presence as the feeling which a learner 
perceives in a certain instructional situation of his/her engagement in learning (Wang & Kang, 2006), people 
should keep in mind that the meaning implies ‘being in classroom, or a place where educational purpose 
(Garrison, 2007) take place’. Also, in order to distinguish presence in learning from many other interpretations 
of presence, a simple descriptor ‘learning’ ahead of ‘presence’ would help the following discussion. 

Constructive Models of Learning Presence 

Among the varying body of studies on presence, there are two important models of presence in specific 
to learning. One is Community of Inquiry Model by Garrison, Anderson, & Archer (2000), and the other is 
Cybergogy Model for engaged learning by Wang, & Kang (2006). Both models present three components of 
presence, but they differ in several aspects. 
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Community of Inquiry Model 



In the earlier study of Garrison, Anderson, & Archer (2000), they suggested a model of Community of 
Inquiry (Col), consisting of three components: Cognitive presence, social presence, and teaching presence 
(Figure. 1). In this model, cognitive presence is the most core and fundamental aspect of all three, and it aims at 
critical thinking as a final destination for effective learning. Social presence acts as a supporting agent in the 
process indirectly facilitating with help from teaching presence. Here, teaching presence has functions of 
designing and facilitating cognitive presence. Moreover, the author adds “the responsibility to design and 
integrate the cognitive and social elements for educational purposes (p.OO)” to teaching presence. All of these 
imply a kind of hierarchical structure under the theoretical framework, by the use of term facilitation. For 
instance, social presence facilitates cognitive presence, and teaching presence facilitates the integration of 
cognitive and social elements. 




Communication Medium 



Figure 1 . Elements of an educational experience (Garrison, Anderson, & Archer, 2000) 

Although their model explains much about an educational experience as a whole, the focus in the 
model is somewhat distributed to learners and their teacher, as the underlying assumption of the model is that 
the Col is “composed of teachers and students (p.OO).” It covers the dynamics of classroom interaction, both 
learning and teaching, by including teaching presence alongside other presences related mostly to learning. 

From the discussion, thus, it is probable to conclude that the model is concerned with the interpersonal aspects 
among the members sharing a same educational experience, which are also the members of Col. 

In regard to teaching presence, the article says it can be “performed by any one participant (p.OO)” in 
the community. The fact that these authors specifically mentioned ‘any’ here brings an important implication to 
the model. As a Col bases on computer-mediated communication (CMC), there is no predetermined learner and 
teacher. That is to say, learners can act as a teacher and together they construct knowledge. Unfortunately, 
inconclusive deduction to the previous theoretical discussion is found in the functions mentioned as of teaching 
presence, design and facilitation, in that they are mainly the role of traditional teacher. Perhaps facilitation 
function can be performed by learners, as in facilitating learning through cooperating with others. Nonetheless, 
the former function, design, is much difficult to be carried out by learners since the term implies explicit, direct 
method of instruction. In fact, the authors also added in the article; “however, in an educational environment, 
these functions (of teaching presence) are likely to be the primary responsibility of a teacher (p.OO).” 

Cybergogy Model for Engaged Learning 

In 2006, a different approach to presence of learning has been studied by Wang & Kang. Though the 
shape and some of the components may seem similar, the originality of each component is sustained by 
distinctive characteristics of the channel through which these components are perceived. In this model, teaching 
presence in the previous model has been replaced by emotive presence (see Figure 2). Kang (2005) stated in 
regard to the reason for adopting emotive factors rather than teaching that the concept of Cybergogy already 
bears the meaning of instruction within. Thus, it is noticeable that the model targets solely the learner as 
participant, and it suggests engaged learning as a core of learner’s learning experience. 

In this model, learners are supposed to engage in deep learning, an ideal status for effective learning, 
when all three presence levels reach at a certain height. As a matter of fact, the concept of learning presence 
here has developed on two basic assumptions that the engaged learning happens when three components of 
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presence coexist and that the engaged learning promises positive learning results. Schlecty (1994) once noted 
about engaged learning that “students are engaged when they are attracted to their work, persist despite 
challenges and obstacles, and take visible delight in accomplishing their work (p.OO).” Bomia et al. (1994) also 
tried to explain engaged learning as referring it to "student's willingness, need, desire and compulsion to 
participate in, and be successful in, the learning process (p.OO).” As mentioned, engaged learning is a state of 
learner that he/she is fully involved in the process of learning experience. This, in turn, will lead the learners to 
effective learning results. 




Figure 2. Cybergogy for engaged learning: Increasing the level of presence (Wang, & Kang, 2006) 

One unique aspect of the model is that it tried to emphasize the role of emotive factors in learning. 
Along with behaviors and cognition, emotions have been a major aspect in study of human learning, but it was 
not until recently that the research on emotional factors have gained proper spotlight. Emotions and feeling 
together can be grouped into affective matters in human psychology and they are often characterized as more 
obscure, unstable, and subjective variables. People find them hard to measure or numerate, therefore 
complicating the process of exploring (Kang, Kim, & Choi, 2007). However, Campbell and Cleveland-Innes 
(2005) indicates a need for a possible distinction between social and emotional aspects in presence. 

Summary 

In summary, the following table briefly outlines the comparison of the two models. 





Community of Inquiry model 


Cybergogy model 


Basis 


Educational experience in 
virtual environments 


Art of education in cyberspace 


Core 


Educational experience 
(i.e. critical thinking) 


Engaged learning 


Components 


• Cognitive presence 

• Social presence 

• Teaching presence 


• Cognitive presence 

• Social presence 

• Emotive presence 


Learner’ s 
Surroundings 


Communication media 


Online learning environment 


Acting 

Participants 


• Learners 

• Teachers 


• Learners 


Assumption 


Systematic facilitation from 


Coexistence of all three 


for effective 
learning 


teaching presence toward 
cognitive presence 


components 


Focus 


Concerned with the 
interpersonal aspects among the 
entire members sharing an 
educational experience. 


Concerned with the 
intrapersonal aspects within the 
learner. Other agents in the 
educational situation is 
considered ‘environmental’ 



Table 1. Summary of the model comparison: Community of Inquiry model vs. Cybergogy model 
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First of all, the components of the models differ. Garrison, Anderson, & Archer (2000) suggested a 
model of Community of Inquiry (Col), consisting of three component cognitive presence, social presence, and 
teaching presence. In this model, cognitive presence is the most core and fundamental aspect of all three, and it 
aims at critical thinking as a final destination for effective learning. Social presence acts as a supporting agent 
in the process indirectly facilitating with help from teaching presence. Here, teaching presence has functions of 
designing and facilitating cognitive presence. It explains the role of teacher and some part of instructional 
designer. 

A different approach to presence of learning has been studied by Wang & Kang (2006). Though the 
shape and two components are similar to Col model, Cybergogy model takes in emotional presence rather than 
teaching presence. On the surface, the teaching presence in the previous model has been replaced by the 
emotive presence. However, the reason for adopting emotional presence is that emotion itself became more and 
more important and that the model is basically focused on learners. 

Second, the Col model includes both the teacher and the students as participants and deals with the 
entire process of learning, whereas the Cybergogy model focused solely on the learner and their status during 
learning. As seen above, the Col model covers teaching presence which mostly is conducted by teacher. Also, 
the authors suggest that the center of the model is educational experience. These two aspects together tell that 
the Col model represents broader range of education as an experience. On the other hand, the Cybergogy model 
targets only the learner as participant. In addition, the core of the model is engaged learning, which is the status 
of the learner. In short, the Cybergogy model describes the learner’s status during the incident of learning, in 
three aspects, cognitive, social, and emotional presence, respectively. 

Third, the Col model tends to show systematic evolvement in each component of presence, while in the 
Cybergogy model each component of presence simply coexists and tends to show interrelation. For instance, 
according to the Col model it is hardly possible to expect cognitive presence if there is no sign of social or 
teaching presence. However, the Cybergogy model assumes relations or some overlaps among the components, 
but it does not require one to precede one another. Different approach to the constructs of presence leads to a 
different approach in methods of research. 

Empirical Studies of Learning Presence 

After first presenting their constructive model of presence. Garrison, Anderson, and Archer (2000) 
proceeded with developing appropriate measuring instrument for the three constructs of presence, with a 
doctoral student Rourke (Garrison, Anderson, & Archer, 2001; Rourke, Anderson, Garrison, & Archer, 2001; 
Anderson, Rourke, Garrison, & Archer, 2001). As their research had been originated from computer mediated, 
text-based contexts, the methods suggested in their work were majorly focused on quantitative content analysis. 
In quantitative content analysis, qualitative content data samples are cut, coded and counted as quantitative 
frequency. 

These works was based on Rourke, Anderson, Garrison, and Archer (2000). After reviewing 19 
previous work of content analysis, the authors summarized four steps of content analysis: 1) identifying the 
representative samples; 2) creating a protocol; 3) comparing for reliability; and 4) analyzing the data. The 
validity of the analysis result according to these authors should attribute largely to objectivity, reliability, 
replicability and systematic coherence. The authors also stated that even though experimental content analysis 
allows deeper investigation of the context, much of the content analysis are done in descriptive way where only 
simple frequencies are dealt with. Then the discussion moves onto the nature of content, whether manifestative 
or latent. Practical concerns on the decision of unit of analysis were mentioned, summarizing various units into 
sentence, paragraph, message, thematic, illocutionary categories. Later in Anderson, Rourke, Garrison, and 
Archer (2001), the authors added that the most frequently used type of unit is thematic, though message unit is 
easier to measure. 

On the other hand, researchers of Cybergogy model developed a set of student self-report 
questionnaires (Kang, Choi, & Park, 2007; Kang, Park, & Shin, 2007; Kang, Kim, & Park; 2007). Despite some 
commonly known caveats that unreliable answers and a biased selection of offences and interviewees may occur 
self-report method have been highly invited to educational studies due to its wide range of applicability. 
However, to avoid possible criticism, the process of developing questionnaires had undergone a thorough 
examination of checking validity and reliability. Until now, the research team, supported by 2008 BK(Brain 
Korea)21 project in Korea, have finished analyzing two rounds of data in pursuit of developing and validating 
the instrument. The first round targeted 418 undergraduate students and total of 78 presence questionnaires (27 
cognitive, 27 social, 24 emotional items) were asked. The second round of survey was conducted to 305 
undergraduate students with the revised 55 presence questionnaires (18 cognitive, 19 social, 18 emotional items) 
Further revision of the instrument is subject to change based on scientific evidence. Reliability of the presence 
questionnaires were calculated in Cronbach a, with the first being .93 and the second being .89. 
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Cognitive Presence 



Interest had been raised in investigating the role of cognitive presence and its relationship with 
outcomes of learning. Thanks to Information Processing theory and other cognitive learning theories, 
researchers often hypothesize the development of or its the effect on learning outcome. 

In examining the developmental process of cognitive presence. Garrison, Anderson and Archer (2001) 
sampled three transcripts from two graduate level courses. Two graduate students coded the transcripts and the 
interrater reliability was accounted for. Four descriptors (evocative, inquisitive, tentative, and committed) of the 
four phases of practical inquiry model (triggering event, exploration, integration, and resolution) were 
constructed to analyze the texts. Using messages as the unit of analysis, the first two transcripts included 51 
messages of 14 people and 20 messages of 7 people, and these were used to refine the protocol during the 
process of coding. The third transcript included 24 messages of 5 people and this used the final version of the 
modified protocol. 

The result from the third transcript showed 8 % of triggering event, 42% of exploration, 13% of 
integration, 4% of resolution and 33% of others. Though relatively lower amount of triggering event and 
integration phases is explainable in that triggering event is most likely pre-framed by the teacher and that 
integration is a challenging process that requires reflection and synthesis. The authors conclude the discussion 
with noting that relatively low amount of resolution phase might have came from lack of advanced inquiry in 
the instructional design and facilitation itself, an inappropriateness of the medium kind, and an 
inappropriateness of the research framework, the practical inquiry model. 

Quantitative cognitive presence data based on Cybergogy model showed positive relationship between 
learners’ cognitive presence level and their academic achievement and satisfaction (Kang, Kim, & Kang, 2008; 
Kang, Kim, & Park, 2008). The instrument used for measuring cognitive presence was originally constructed 
based on a conceptual framework of general understanding, knowledge construction, and learning management. 
Through a factor analysis process, these constructs were confirmed (Kang, Park, & Shin, 2007). Kang, Kim, 
and Kang (2008) divided the process of learning into before (self-efficacy, metacognition), during (cognitive 
process, flow), and after (achievement, problem-solving performance, satisfaction) factors. The authors tested 
the relationship among the factors with 60 undergraduate students. The result showed a predictive power of 
cognitive presence toward after learning variables. 

Social Presence 

As computer mediated world lacks physical relationship among learners and between learner and 
teacher, the first major concern of researchers in this field was geared at how to promote social presence of 
learners. Social presence is mostly based on peer interaction, but occasionally wider interpretation is possible to 
encompass teacher roles. 

In assessing the social presence in computer mediated environment, Rourke, Anderson, Garrison, and 
Archer (1999) proposed three dimensions of social presence: affective, interactive, and cohesive responses. 
These three dimensions, or categories which was originally used in the article, include a total of 12 indicators. 
For instance, they suggest affective category to have expression of emotion, use of humor, and self-disclosure. 
Using thematic units, two sample transcripts showed a distinctive result. Two transcripts were outcomes of two 
one-week discussion sessions, and the numbers of participants were 14 and 17, respectively. By using thematic 
units, the total number of messages and instances (messages cut into thematic units) differed greatly, between 
Transcript A and B. To adjust this problem, the researchers go through several calculating process. Firstly they 
sum the raw number of instances, divided it by the total number of words, and finally multiply 1000. These new 
values are named “social presence density” per 1000 words. Social presence density for Transcript A and B was 
22.83 and 33.54, respectively. Interrater reliability was also shared (Transcript A: .95, Transcript B: .91). The 
result of the analysis introduces the glimpse of the social aspect of learning process. 

Nevertheless, this study simply stays at a descriptive level of analysis. The applicability of the social 
presence studies would increase after unveiling its relationship with outcomes of learning. In this regard, Kang 
and Kang (2008) investigated the relationship of social presence with satisfaction and learning outcomes in a 
blended learning context. The data was collected from 46 undergraduate students, and the result implied that the 
social presence could be the predictor of learning outcome. This result supports the importance of promoting 
social presence of learners to increase their learning outcome. 

However, this positive relation is yet unclear. In Kang, Park, & Choi (2006), the authors tried to 
measure the predictive power of prior knowledge and social presence toward satisfaction and achievement in 
online teacher training. Data from 71 in-service teachers was collected on prior knowledge (IT ability. Flash 
ability) and social presence (interaction, cohesiveness) during the course, and satisfaction and achievement data 
was collected after the course. Result showed only IT ability and cohesiveness predicted satisfaction. 
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Teaching Presence 

In a context where the role of the teacher is a major independent variable of success, identifying the 
key factors in teaching process becomes equally important. Anderson, Rourke, Garrison, and Archer (2001) 
assessed teaching presence in a computer conferencing context. Having three sub-categories of teaching 
presence (design and organization, facilitating discourse, direct instruction), a content analysis by message unit 
was conducted on two courses of 13 week conference transcripts. Two raters coded the transcript and the result 
showed proportional data of teacher’s message coded by the three indicators mentioned above. The most 
frequent data in both courses were direct instruction, 77% and 87.5%, respectively. The second most was 
facilitating discourse (43.2%, 75.0%), and the last was instructional design (22.3%, 37.5%). However, some of 
the reviewers criticized on using only the during-course data to measure teaching presence, in that many 
teachers spend a considerate amount of time before the course begins. The authors conclude the discussion with 
emphasizing the importance of further studies in the field of teaching presence. In fact, there is a worldly 
recognition in the role of teacher as the learning environment advances. 

Emotional Presence 

Not many studies investigated the influence of emotional presence in learning. However, Kang, Kim 
and Park developed an emotional presence scale in 2007. The scale had three sub-categories: Feeling, 
expressing, and managing emotion. Among the three constructs of Cybergogy model, emotional presence is the 
most unique and yet unstable construct. Recent interest in the importance of emotion in learning supports the 
idea of suggesting emotional presence as a separate construct from social presence. The few existing data 
analysis shows that emotional presence predicts satisfaction, but hardly achievement of learning (Kang, Kim, & 
Park, 2008; Kang, Suh, & Moon, 2005). 

Summary 

In summary, the following table outlines the research approaches and suggested indicators of the 
constructs of the two models. Sub-categories of each constructs are excerpted from the previous work of both 
Col and Cybergogy research parties (Anderson, Rourke, Garrison, & Archer, 2001; Garrison, Anderson, & 
Archer, 2001; Kang, Choi, & Park, 2007; Kang, Kim, & Park, 2007; Kang, Park, & Shin, 2007; Rourke, 
Anderson, Garrison, & Archer, 2001). 



Community of Inquiry model Cybergogy model 



Approach 


• Quantitative content analysis 

• Unit of analysis: thematic, message . . . 

• Introduction of the concept ‘social presence density’ 


• Self-report questionnaire 

• Two rounds of data collection 

• Factor analysis 




Triggering 

Event 


• Recognizing the problem 

• Sense of puzzlement 




Cognitive 


Exploration 


• Divergence — within the online community 

• Divergence — within a single message 

• Information exchange 

• Suggestions for consideration 

• Brainstorming 

• Leaps to conclusions 


General ' 

Understanding 


• Consistency between content 
and objective 

• Organization of content 

• Articulation of content 




Integration 


• Convergence — among group members 

• Convergence — within a single message 

• Connecting ideas, synthesis 

• Creating solutions 


Knowledge 

Construction 


• Information acquisition 

• Information transformation 

• Knowledge construction 




Resolution 


• Vicarious application to real world 

• Testing solutions 

• Defending solutions 


Learning 

Management 


• Time management 

• Performance management 

• Environment management 




Affective 

Responses 


• Expression of emotions 

• Use of humor 

• Self-disclosure 


Co-presence 


• Not being isolated 

• Mutual awareness 

• Mutual attention 


Social 


Interactive 

Responses 


• Continuing a thread 

• Quoting from others’ messages 

• Referring explicitly to others’ messages 

• Asking questions 

• Complimenting, expressing appreciation 

• Expressing agreement 


Influence 


• Mutual understanding 

• Mutual interdependence 

• Mutual assistance 
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Cohesive 

Responses 


• Vocatives 

• Addresses or refers to the group using inclusive 
pronouns 

• Phatics, salutations 


Cohesiveness 


• Positive interaction 

• Chances of contribution 

• Sense of group commitment 




Instructional 
Design and 
Organization 


• Setting curriculum 

• Designing methods 

• Establishing time parameters 

• Utilizing medium effectively 

• Establishing netiquette 






Teaching 


Facilitating 

Discourse 


• Identifying areas of agreement/disagreement 

• Seeking to reach consensus/understanding 

• Encouraging, acknowledging, or reinforcing 
student contributions 

• Setting climate for learning 

• Drawing in participants, promoting discussion 

• Assessing the efficacy of the process 




- 




Direct 

Instruction 


• Present content/questions 

• Focus the discussion on specific issues 

• Summarize the discussion 

• Confirm understanding through assessment and 
explanatory feedback 

• Diagnose misconceptions 

• Inject knowledge from diverse sources 

• Responding to technical concerns 












Feeling 

Emotion 


• Comfortableness 

• Security 

• Interest 


Emotional 




- 


Expressing 

Emotion 


• Freedom 

• Diversity 

• Clarity 








Managing 

Emotion 


• Flexibility 

• Activeness 



Table2. Research approaches and indicators of the two models 



Implications and Suggestions 

Although there are increasing number of research initiatives around the world, presence research in 
learning contexts is still at its enfant stage. It should be noted that the results from prior work provides only a 
partial accounts for the entire process of learning. 

The major implication from the current comparative analysis is the notion that different models can he 
used to describe different situations, with different purposes. For instance, if the research question lies solely on 
the status of the learner, the proper model for this investigation would be Cybergogy model. On the other hand, 
if the research question concerns both the learner and the teacher, then the appropriate model to refer to would 
be Community of Inquiry model. Furthermore, different research methods suggested indicates a possibility of a 
synthesis in the presence study. Both quantitative content analysis and the self-report questionnaire can he 
implemented in a single research setting, supplementing the weakness of each method. 

Three suggestions for the future study are made by the authors. First, more of the future studies on 
learning presence should be conducted experimentally and designed to measure the change of presence level. 
Since the concept of presence began as an unstable, inconsistent variable, this kind of approach would provide a 
wider view of the concept. For example, the purpose of the study should ask whether a certain educational 
treatment has affected the learner’s presence positively or negatively. Second, regarding the context of the 
presence research, applications should be made to cover not only online classrooms but also blended learning, or 
better yet offline classrooms. This trend of widening the research premises is inevitable as the technology 
becomes more intuitive and ubiquitous. In fact, recent studies were conducted to blended learning environments 
(Kang, Kang, & Jung, 2008; Kang & Kang, 2008). Third, as presence being abstract and metaphysical concept, 
research on presence should consider a collaborative work with neuro-science researchers. The advancement in 
neuro-technology enables researchers to gain direct feedback from human brain. In addition, the applicability of 
the neuro-scientific method would enhance as the abstract nature of the construct increases, as in case of 
emotional presence. Emotion is indeed a tricky concept to handle. As stated in Picard et al. (2004) previous 
understanding of emotion was that too much is bad for rational thinking, but recent findings say the same thing 
to too little emotion. This implies that there is a possible ground to hypothesize a curving relation between 
emotion and academic achievement. Whether true or not is left undiscovered. 
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Abstract: This study investigated the relationship among learning authenticity, motivation, and achievement in 
web-based project learning. Sixty-eight undergraduates who attended a web-based project were participants. 
Following are the results: 1) Learning authenticity predicted motivation (P = .672, p < .05) and achievement (P 
= .554, p < .05), 2) Motivation predicted achievement (P = .583, p < .05). 3) Motivation mediated the relationship 
between learning authenticity and achievement. More specifically, task value and self-efficacy were two mediating 
sub-variables under motivation where resource authenticity predicted perceived achievement. Implications for 
enhancing learning authenticity and suggestions for future research were discussed. 

Keywords: learning authenticity, motivation, achievement, web-based project learning 

Introduction 

Constructivists are challenging the traditional learning environment insisting to meet the needs of real-life and 
provide the opportunities to explore ill-defined problems. A web-based project learning environment could be in line 
with constructivism where learners struggle to complete the given project using various learning resources on the 
web, interacting with their peers and experts synchronously and asynchronously. This open and flexible learning 
environment presents new challenges for instructional designers to create an authentic learning environment where 
students freely explore, share, and generate ideas and deliverables (Zembal-Saul et al., 2002). 

Learning authenticity here is defined as a level of students’ perceived authenticity on how realistic of the 
learning process. Since learning authenticity is a multi-faceted factor, it does not exist in the learner, the task, or the 
environment itself, but in the dynamic interactions among these components (Barab, Squire, & Dueber, 2000; 
Guilkers, Bastiaens, & Kirschner, 2004; Herrington, Oliver, & Reeves, 2003; Hung & Chen, 2007). In spite of the 
complex aspect of learning authenticity, previous studies treated this variable very simple and mainly focused on 
task authenticity. Yet, the results were not clear on the relationship between task authenticity and learning outcomes 
(Hung & Chen, 2007; Jo & Lim, 2002). In order to proceed further in the research area of learning authenticity, not 
only the dimensions of multi-faceted authenticity but empirical validation on the impact of learning authenticity 
should be tested rigorously. 

Previous research indicated the possible relationship between learning authenticity and motivation. Petraglia 
(1998) advocats the positive relationship stating that learning authenticity itself contributes to the promotion of 
learning motivation without any extra effort of motivational intervention. Students acquire knowledge that meets 
their practical needs and explore the solutions to questions connected directly to their real-life experience. They 
actively participate in learning experience if high level of motivation is enhanced by learning authenticity (Schwartz, 
Brophy, & Bransford, 1999). Research, however, reports conflicting results on the relationship between authenticity 
and motivation. For example, some research report that students persevere their learning with high learning 
authenticity (Ames, 1992; Hung & Chen 2007; Kang & Kim, 1999; Petraglia, 1998; Park & Kim, 2004), but others 
report that learners who work in an authentic environment did not perform better than those who are in a less 
authentic environment (Gulikers, Bastiaens, & Martens, 2005). 
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To identify the relationship between learning authenticity, motivation, and achievement, the present study 
examined a web-based project learning context which requires learners’ highly-motivated. Research problems set for 
the research are: 

1 ) Does learning authenticity predict motivation and achievement? 

2) Does motivation predict achievement? 

3) Does motivation mediate the relationship between learning authenticity and achievement? 

Theoretical Background 



Learning Authenticity 

Learning authenticity which is defined as students’ perception on learning process and environment has 
potential to foster meaningful learning (Petraglia, 1998). Project learning with high level of learning authenticity 
encourages students in investigation through an inquiry process structured around complex, authentic tasks (Buck 
Institute for Education, 2001). Given an authentic task. Students pursue solutions by testing their hypothesis, 
designing action plans, negotiating their ideas, making predictions, collecting and analyzing data, and drawing 
conclusions (Blumenfeld et ah, 1991). Moreover, in a web-based project learning environment, students are able to 
access information easily, interact with other peers, utilize various resources, and elaborate their knowledge based on 
asynchronous collaboration. Learning authenticity, therefore, could be composed with four areas such as learning 
activities, tasks, resources and assessment. 

Authenticity of activities 

Constructivism has placed the students’ activity at the heart of the curriculum (Reeves, Herrington, & Oliver, 
2002). Within this philosophy, research suggests that authentic activities provide the opportunity for students to 
approach the task from different perspectives through collaboration (Herrington, Oliver, & Reeves, 2003; Roelofs & 
Terwel, 1999). Honebein and his colleagues (1993) argued that learning authenticity can be enhanced by project 
learning which entails activities and sub-activities required to complete the project. Similarly, Roelofs and Terwel 
(1999) distinguished authentic pedagogy such as cooperation and communication that is based on Newmann, Marks, 
and Gamoran (1996). These views imply that learners’ authenticity of activities can be fostered when they engage in 
the collaboration with share of information, interaction for examining different perspectives, and deep reflection for 
negotiating their thought. 

Authenticity of tasks 

Complex tasks with proper level of challenges and realistic situations could be one dimension of learning 
authenticity (Herrington, Oliver, & Reeves, 2003; Honebein, Duffy, & Fishman, 1993; Kang & Kim, 2000; Petraglia, 
1998; Roelofs & Terwel, 1999). First of all, authentic tasks should be connected to students’ personal worlds 
(Guilkers, Bastiaens, & Kirschner, 2004; Roelofs & Terwel, 1999). Students may encounter realistic cognitive 
conflict with tasks. Complex tasks also include genuine constraints such as deadlines and time allowances 
(Herrington & Oliver, 2000). Students perceive tasks as authentic if they examine the real issues of tasks and look 
for alternatives over a sustained period of time. 

Authenticity of resources 

By richness of learning resources, web-based learning environments can be distinguished from traditional 
learning environments. Authentic learning environments take advantages of rich resources. Multiple resources allow 
students to utilize practical resources by expanding them beyond texts to such sources as schedules, maps, and charts 
(Herrington, 2005; Rule, 2006). In order to construct final products, students capture rich information, manipulate 
enormous information, and reinterpret that information. Students can leverage quality of their outcomes with 
resources that practitioners may employ. Authenticity of resources based on web technologies may foster learner’s 
active learning and diminish learners’ perceived gap between learning and practice. 

Authenticity of assessment 

Recent research emphasize that authentic learning should be seamlessly integrated with assessment (Guilkers, 
Bastiaens, & Kirschner, 2004; Herrington, 2005; Herrington, Oliver, & Reeves, 2003; Kim, 2007; Woo et ah, 2007). 
It is important that authentic assessment should resemble the social processes in reality and integrated into learning 
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process (Guilkers, Bastiaens, & Kirschner, 2004). Gulikers and his colleagues (2004) identified five dimensions of 
authentic assessment: the physical context, the social context, the assessment of tasks, the assessment of result, and 
the assessment of criteria. These dimensions should be integrated to learning process. 

Motivation 

Learning from a realistic and complex project requires learners’ high level of motivation. If students are not 
motivated to learn, they are less likely to invest the time in their learning and thus will not benefit from its learning. 
Reversely, learners with high level of motivation can participate actively in learning experience. Learning process 
therefore promote motivation by providing the authenticity of learning (Schwartz, Brophy, & Bransford, 1999; 
Petraglia, 1998). 

Expectancy and value components of motivation are important for predicting students’ actual achievement 
(Pintrich & Schunk, 2002). Four proximal factors were defined based primarily on an expectancy- value model by 
Pintrich and his colleagues (1991): goal orientation and task value as value components and attribution and self- 
efficacy as expectancy components. These are the factors to be considered in learning authenticity. 

Goal orientation 

Goal orientation refers student's general orientation to the course as a whole (Pintrich, Smith, Garcia, & 
McKeachie, 1991, p. 9) and how learners approach and engage in learning tasks (Ames, 1992; Pintrich & Schunk, 
2002). Two goal orientations, mastery and performance goal, are generally mentioned in recent studies. Mastery 
goal orientation focuses on learning new skills, improving or developing competencies, and trying to gain 
understanding or insight for self-improvement. Performance goal orientation rather focuses on demonstrating ability 
relative to others and striving to be the best in the classroom. Research reported that goal orientation is strongly 
related with achievement (e.g. Bong, 2008; Hulleman, Durik, Schweigert, & Harackiewicz, 2008; Roebken, 2007). 

Task Value 

Task value refers to students' perception on how interesting, how important, and how useful the task is 
(Pintrich et ak, 1991; Wigfield & Eccles, 1992). Earlier cognitive perspective on motivation stressed expectancy of 
success and value of tasks which various motivation theories were based on (Pintrich & Schunk, 2002). If students 
don’t value the task, they will be less likely to choose to engage in task, consequently, they will not gain knowledge. 
Reversely, the more students perceive the importance and meaningfulness of tasks, the more they persist at learning 
and acquire knowledge, not giving up when they faced with failure. For example, Hulleman, Durik, Schweigert, & 
Harackiewicz (2008) examined the antecedents and consequences of task value in two learning contexts: a college 
classroom and a high school sports camp and the result showed that task value predicted actual performance. 

Attribution 

Weiner postulated that students attribute success or failures of their outcomes to factors such as ability, effort, 
task difficulty, and luck (Zimmerman & Schunk, 2007). In attribution theory, learning outcomes are determined by 
causal dimensions (Pintrich & Schunk, 2002). When students believe that their effort to learn will result in positive 
outcomes, they are more likely to study more strategically and effectively (Pintrich et ak, 1991). For example, 
Uquak, Elias, Uli, & Suandi (2007) showed that people who attribute their own failure to their own ability and 
efforts can recover failures and move on to adaptive ways of coping and achievement. 

Self-efficacy 

Self-efficacy refers to judgments of personal capabilities to organize and execute courses of action required to 
attain designed types of goals (Bandura, 1997). Self-efficacy are contextually specific in that they refer to specific 
performance situations depended on properties such as the conditions of learning or an attribute of tasks 
(Zimmerman & Schunk, 2007). Previous studies reported self-efficacy as representative motivation on achievement 
(Bandura, 1997; Bong, 2008; Pintrich & Schunk, 2002; Zimmerman & Schunk, 2007). For example. Bong (2008) 
reviewed the importance and socio-cultural factors of self-efficacy, goal orientation, task value, and attribution 
related to adolescents' cognition, affect, and behavioral patterns in achievement settings. She contented the link 
between self-efficacy and Korean adolescents' academic achievement. 
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Relationships between learning authenticity, motivation, and achievement 



Although few studies have addressed the link between learning authenticity and motivation, some research 
suggested that learning authenticity promoted learners’ motivation (Ames, 1992; Hung & Chen 200; Kang & Kim, 
1999; Petraglia, 1998; Park & Kim, 2004). Ames (1992) contended that classroom structures including tasks 
contribute to student’s goal orientation. Also, whether students are oriented to their goals has consequences for 
whether they try hard and take on challenges. He suggested that task, assessment and recognition, and authority 
dimensions of classrooms are presented as examples of structures that can influence children's orientation toward 
different achievement goals. Kang and Kim (2000) found that there was a significant difference in the learner's 
motivation (ARCS) by the levels of authenticity. Hung and Chen (2007) implied the importance of authenticity of 
task in that when the tasks are interesting to the learners and related to their life, learners can participate in solving 
the tasks. Park and Kim (2004) indicated that there was the improvement of learners’ self-efficacy and interest by 
utilizing authentic resources. 

Learning authenticity, also, is thought to serve as a predictor of achievement (Kang & Kim, 1999; Kim, 2007; 
Elliot & Langlois, 2002; Woo et ak, 2007). Elliot and Langlois (2002) showed that action research implementing 
authenticity of activities could improve students’ learning and diminish achievement gaps among low-progress and 
high-progress learners. Kim (2007) suggested that authentic assessment and immediate feedback focusing on 
students' learning process have a positive influence over students’ products. In Woo and her colleagues (2007), 
students reported that the authentic tasks fostered positive feelings of learning and achievement. 

Research on motivation reported that motivation was a strong predictor of achievement (Bandura, 1997; Hill 
& Hannafin, 1997). Hill and Hannafin (1997) supported the link between self-efficacy and achievement by findings 
that perceptions of self-efficacy influenced the strategies learners used. Oh (2003) examined relationship between 
sex-role identity, locus of control and academic achievement level. The results indicated that those who attributed 
their success to internal locus of control had a higher academic achievement level. 

Mediating role of self-efficacy between task complexity and students’ performance were identified by 
Kitsantas and Balyor (2001) and Mangos and Steele-Johnson (2001). Eor example. Mangos and Stelle-Johnson 
(2001) identified that the effect of subjective task complexity on performance was mediated by self-efficacy in a 
computerized simulation of a class scheduling task. 



Methods 



Participants 

Participants were 68 undergraduate students who enrolled in introductory Educational Technology course at a 
university in Korea. Participants were randomly assigned to one of fourteen teams to perform a project 
collaboratively. 

Context 

They conducted a project in Cyber Campus, given the task to investigate Educational Technology in practice. 
Based on the process of Jo (1999) and Lee (2001), the project lasted for five weeks as they planned, conducted, 
presented, and assessed their project collaboratively. Eirst, students planned to do their project assigning their roles 
and questioning about solve the task, creating a timeline, and determining the required resources. In conduct phase, 
they carried out their action plans by exploring related theories and methodology and then, analyzing cases in 
practice. In addition, they visited and interviewed the practitioners. In the presentation phase, they produced final 
outcomes with multimedia and exhibited them. In the assessment phase, they evaluated other team products. Also, 
students submitted a short paper on a topic out of other team individually. After completing project, all participants 
were asked to answer learning authenticity, motivation, and perceived achievement questionnaire. 

Measures 

Based on theoretical review of learning authenticity (Gulikers, Bastiaens, & Martens, 2004; Herrington, Oliver, 
& Reeves, 2003; Herrington, 2005; Honebein, Duffy, & Eishman, 1993; Roelofs & Terwel, 1999) and the scale of 
authenticity of tasks (Kang & Kim, 1999), four factors were identified by an EEA (Exploratory factor analysis). 
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Principal axis factoring method was used to extract factors. To rotate factors, direct oblimin rotation method was 
used. Scree test with visual inspection was also used to determine the number of factors to be extracted. We labeled 
the four factors as: activity authenticity, task authenticity, resource authenticity, and assessment authenticity. Each 
question was scored on a 5-point Likert Scale. The reliability of fourteen items was Cronbach a = .88. Some sample 
statements used for this subscale were: “I exchanged opinions with other participants while still maintaining a sense 
of respect.” for activity authenticity, “I think task was valuable and meaningful to me.” for task authenticity, “I 
utilized a variety of sources to solve the task.” for resource authenticity, and “When given task, criteria for 
assessment was provided.” for assessment authenticity. 

Motivation items were adapted and modified from MSLQ (Pintrich et ah, 1991). It consisted of 26 questions 
that measured four factors; goal orientation, task value, attribution, and self-efficacy. The reliability of items was 
Cronbach a = .88. Each question was scored on a 5-point Likert Scale. Some sample statements used for measuring 
the learning strategy were: “Getting a good grade in this class is the most satisfying thing for me right now.” for goal 
orientation, “It is important for me to learn the course material in this class.” for task value, “If I don't understand the 
course material, it is because I didn't try hard enough.” for attribution, and “I'm certain I can understand the most 
difficult material presented in this course.” for self-efficacy. 

Achievement was measured by four aspects: discussion, perceived achievement, individual and team 
performance. Discussion. All messages posted by students on the discussion board during the project were coded 
based on Lung’s 4 categories (Lung, 2004): academic, building relations, support, and appreciation. One of author 
and a research assistant counted the total number of messages and rated them on a 5-point Likert Scale. Inter-rater 
reliability was .92. Individual and team performance .Two raters worked on analyzing individual and team 
performance. They adapted and modified a set of detailed criteria from previous studies (Kang & Kim, 2002; Kang 
& Kwon, 2004; Kwon, 2002): logicality, organization, and creativity for individual performance and 
accomplishment, relatedness, logicality, and creativity for team performance. They rated them on a 5-point Likert 
Scale and inter-rater reliability was .91. Perceived achievement. Items measuring perceived achievement with five 
items were adapted and modified from Shin (2003). The reliability of modified items was Cronbach a = .74. Some 
sample statements used for measuring the achievement were: “I mastered knowledge and skill from learning”. 

Results 

Collected data were analyzed using Pearson’s correlation and multiple regression. Significant level for all 
results was set at .05. 

Correlation analysis 

Correlations among all of the measurements are analyzed with Pearson’s r coefficient. The result showed that 
learning authenticity was significantly positively related to motivation (r = .67, p < .05) and achievement (r = .55, p 
< .05). Also, motivation was found to be correlated significantly positively with achievement (r = .58, p < .05). 

Correlations for sub-variables are presented in Table 1 . The result indicated positive correlation between task 
authenticity and task value (r = .62, p < .05). Resource authenticity was significantly related to self-efficacy (r = .60, 
p < .05). Also, task value was found to be correlated significantly with perceived achievement (r = .70, p < .05) and 
self-efficacy correlated significantly positively with perceived achievement (r = .66, p < .05). 

Table 1 Correlations for learning authenticity, motivation, and achievement 

1 2 3 4 5 6 7 8 9 10 11 12 



1 . Activity authenticity 


- 






2. Task authenticity 


.349* 


- 




3. Resource authenticity 


.523* 


.299* 


- 


4. Assessment authenticity 


.549* 


.457* 


.429* 


5. Goal orientation 


.071 


.206 


.151 .061 
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6. Task value 



7. Attribution 

8. Self-efficacy 

9. Discussion 

10 .Individual performance 

1 1 . Team performance 

12. Perceived performance 



. 532 * 


. 621 * 


. 525 * 


. 507 * 


. 260 * 


- 










. 374 * 


.220 


. 423 * 


. 281 * 


.154 


. 519 * 


- 








. 462 * 


. 243 * 


. 603 * 


. 414 * 


.235 


. 615 * 


. 436 * 


- 






.202 


.113 


. 392 * 


. 274 * 


. 347 * 


. 278 * 


.151 


. 376 * 


- 




.118 


.215 


.111 


. 268 * 


.068 


.102 


-.111 


.030 


. 280 * 


- 


.190 


. 465 * 


. 343 * 


. 362 * 


.201 


. 444 * 


. 274 * 


. 403 * 


.237 


. 330 * 


. 492 * 


. 425 * 


. 596 * 


. 438 * 


.199 


. 697 * 


. 423 * 


. 665 * 


. 262 * 


.058 



.561 



* p < .05 



Predicting of learning authenticity on motivation and achievement 

Predicting of Learning Authenticity on Motivation 

The simple regression analysis revealed that learning authenticity accounted for 45.1% of variance in 
motivation, /'’(I, 66) = 54.201, p < .05, and learning authenticity predicted motivation significantly, P = .672 (see 
Table 2). In the multiple regression analyses of sub-variables, first, no statistical significant result was found in 
regression analysis on goal orientation. Next, the result indicated that variables entered into equation accounted for 
55.4% of variance in task value, T’(4,63) = 19.603, p < .05, and task value was significantly predicted by task 
authenticity, P = .435, and resource authenticity, P = .253. Also, resources authenticity was found to account for 
21.5% of variance in attribution, T’(4,63) = 4.302, p < .05, and it was only found to predict attribution significantly, P 
= .298. Lastly, the result showed that resource authenticity accounted for 40.5% of variance in self-efficacy, T’(4,63) 
= 10.718, p < .05, and it predicted self-efficacy significantly, P = .471. 



Table 2 Results of Regression Models for Predicting of Learning Authenticity on Motivation 



Dependent variables/ 
Sub-variables 


Independent 

variables 


B 


SE 


P 


t 


P 




F 


P 


Motivation 


Learning authenticity 


.772 


.105 


.672 


7.362* 


.000 


.451 


54.201* 


.000 


Task value 


Task authenticity 


.871 


.191 


.435 


4.546* 


.000 


.554 


19.603* 


.000 


Resource authenticity 


.437 


.175 


.253 


2.503* 


.015 


Attribution 


Resource authenticity 


.282 


.127 


.298 


2.223* 


.030 


.215 


4.302* 


.004 


Self-efficacy 


Resource authenticity 


.905 


.224 


.471 


4.035* 


.000 


.405 


10.718* 


.000 



p < .05 



Predicting of Learning Authenticity on Achievement 

Table 3 presents a significant predictive relationship between learning authenticity and achievement. The 
result indicated that 30.7% of variance in achievement was explained by learning authenticity, /'’(I, 66) =29.237, p 
< .05, and learning authenticity predicted achievement, P = .554. On relationships among sub-variables, also, the 
result showed that discussion was predicted by resource authenticity, P = .368, no predictor variables was found on 
individual performance, team performance was predicted by only task authenticity, P = .368, and perceived 
achievement was predicted by only resource authenticity, P = .418. 
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Table 3 Results of Regression Models for Predicting of Learning Authenticity on Achievement 



Dependent variables/ 
Sub-variables 


Independent 

variables 


B 


SE 


P 


t 


P 




F 


P 


Achievement 


Learning authenticity 


.147 


.027 


.554 


5.407* 


.000 


.307 


29.237* 


.000 


Discussion 


Resource authenticity 


.744 


.279 


.368 


2.670* 


.010 


.173 


3.290* 


.016 


Team performance 


Task authenticity 


.469 


.154 


.368 


3.040* 


.003 


.288 


6.369* 


.000 


Perceived achievement 


Resource authenticity 


.508 


.136 


.418 


3.724* 


.000 


.450 


12.868* 


.000 



p < .05 



Predicting of motivation on achievement 

The simple regression analysis revealed that motivation accounted for 34% of variance in achievement, /'’(I, 
66) = 33.982, p < .05, and motivation was found to predict achievement, P = .583 (see Table 4). In the multiple 
regression analyses of sub-variables, self-efficacy was only found to predict discussion, P = .305, perceived 
achievement was predicted by in order of task value, P = .456, and self-efficacy, P = .377. No statistical significant 
results were found in multiple regression analyses of predicting team and individual performance. 



Table 4 Results of Regression Models for Predicting of Motivation on Achievement 



Dependent variables/ 
Sub-variables 


Independent 

variables 


B 


SE 


P 


t 


P 




F 


P 


Achievement 


Motivation 


.134 


.023 


.583 


5.829* 


.000 


.340 


33.982* 


.000 


Discussion 


Self-efficacy 


.321 


.152 


.305 


2.111* 


.039 


.214 


4.286* 


.004 


Perceived achievement — 


Task value 


.320 


.079 


.456 


4.058* 


.000 


.576 


21.417* 


.000 




Self-efficacy 


.238 


.067 


.377 


3.552* 


.001 



p < .05 



Motivation as a mediator of learning authenticity on achievement 

The hypothesized mediating role of motivation on the relationship between learning authenticity and 
achievement was tested as specified by Baron and Kenny (1986) (see Table 5). First, in the model 1, learning 
authenticity was found to predict motivation significantly, P = .672. Next, in the model 2 in which tested the effect 
of learning authenticity on achievement without controlling for motivation, learning authenticity was found to 
predict achievement significantly, P = .554. In the mediated model 3 in which learning authenticity and motivation 
are entered into the same equation, motivation controlling learning authenticity was found to predict achievement. 
Through the same analyses for sub-variables, the results showed that task value and self-efficacy mediated the 
relationship between resource authenticity and perceived achievement. 
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Table 5 Results of Regression Models on the Motivation as a Mediator of learning authenticity on achievement 



Independent variables 


Motivation 




Achievement 




Dependent variables 


Model 1 


Model 2 




Model 3 


Learning authenticity 


.672* 


.554* 




.296* 


Motivation 








.384* 




.451 


.307 




.388 


F 


54.201* 


29.237* 




20.606* 



p < .05 



Discussion 

Findings revealed the empirical relationship between the learning authenticity, motivation, and achievement. 
Even though further investigation is required, we can support the effects of learning authenticity and its 
consequences on motivation and achievement. Result showed that resource authenticity as well as task authenticity 
predicted motivation and achievement. It can be inferred that given task that was highly related to their subject, 
students could perceive the value of tasks and, hence, team performance was increased. This inference can be 
supported by Bleumenfeld and his colleagues (1991) that realistic tasks holding more relevance to students' needs 
and experiences can promote students’ learning, because students can relate easily what they are learning to tasks 
and goals they see every day. At the same time, various types of materials and resources could provide students with 
the opportunity to inquire broadly and produce creative outcomes. Besides, high resource authenticity might 
increase perceived achievement in that when students participate actively in their project by capturing required 
information and manipulating multimedia to construct final learning outcomes, they felt the positive of self-efficacy 
and value of tasks by utilizing resources that practitioners may employ. 

On the other hand, contrary to what was expected, the influence of activity authenticity and assessment 
authenticity on achievement was not evidenced for following reason. Activity authenticity and assessment 
authenticity can be lead by students’ active involvedness collaboratively and individually based on positive 
interdependency and individual accountability (Gulikers, Bastiaens, & Kirschner, 2004). However, as Kreijns and 
his colleagues (2002) mentioned, it can be seen that there are various degrees of disappointing collaboration and 
learning performances: low participation rates, inequality of role and accountability, and surface discussion, etc. 
Also, learners’ unfamiliarity and resistance to spontaneous evaluation caused them to fail to assess quality of 
learning. 

Based on the findings, present study suggests some implications for designing strategies that foster learning 
authenticity in web-based project learning. First, authenticity is not only associated with specific component, but 
with all aspects of learning. As mentioned, various types of materials and resources can provide students with the 
opportunity to inquire broadly and produce creative outcomes. Also, although the predictability of activity 
authenticity or assessment authenticity has not been identified, it is important to provide natural opportunities for 
learners to test and refine their ideas and to construct meaningful knowledge collaboratively. Students need to assess 
their thinking about their activities and their products (Blumenfeld et ak, 1991). Tools such as a schedule, checklist, 
and reflection note allow students to reflect and review their learning process by themselves. Second, based on high 
learning authenticity, motivation enables students to persevere with their learning. Hence, interventions aimed at 
promoting self-efficacy enable students to complete their learning. Another way to increase motivation is to 
encourage learners to challenge and synthesize the value of task through interactions with others. 
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Introduction 

Offshore outsourcing refers to those “service and manufacturing activities of U.S. companies that are 
performed in unaffiliated firms located abroad” (Panel of the National Academy of Public Administration, 2006, p. 
xiv). As a strategy to stay competitive, U.S. companies are offshore outsourcing functions which are considered not 
core competencies to their business. It was reported that 19% of all companies have an outsourcing strategy. 
Surprisingly, 95% of Fortune 1000 firms are using it (The Ventoro Institute, 2005). Offshore outsourcing 
information technology (IT) aspects especially in the area of production programming and system maintenance 
programming (Gurbaxani & Jorion, 2005, as reviewed by Laudon & Laudon, 2007) has become one of the latest 
business strategies for corporate America. For example, the Information Technology Association of America 
(IT A A) sponsored a survey to evaluate the comprehensive impact of offshore IT and services outsourcing on the 
U.S. economy and the IT industry (Global Insight Inc., 2004). The survey results reveal that competitive necessity, 
quality gains, and cost savings are the leading driving forces behind the momentum of offshore outsourcing. Despite 
the numerous benefits brought about by IT offshore outsourcing, offshore outsourcing also incurs challenges 
including reduction or loss of technical know-how, knowledge transfer, reduced degree of control, communication 
management, cultural differences (Davison, 2003; Haag & Cummings, 2007). 

Colteryahn and Davis (2004) claim that corporate America is witnessing another transformation, because 
many organizations are offshore outsourcing some of their learning and training functions in the hope of eliminating 
fixed costs, focusing on core competencies, becoming more strategic, and containing high and unpredictable costs of 
enterprise learning. Despite the widely publicized discussions of advantages and disadvantages of offshore 
outsourcing, little research has been conducted on the offshore outsourcing of training and development functions. 
To fill the gap in the literature, a phenomenological case study (Yin, 1994) was conducted to explore how training 
and development professionals perceived their experiences of offshore outsourcing. 

The Corporate Training and Development Group of one of the world’s leading package delivery companies 
adopted the offshore outsourcing strategy five years ago. Components related to the development of training 
materials including computer programming, graphics design, and animation started to be outsourced to one of 
India's largest and most respected business conglomerates. The case study was conducted to investigate 
stakeholders’ perceptions of the offshore outsourcing strategy. The following research questions guided the case 
study: 

1 . What was training and development professionals’ experience in offshore outsourcing? 

2. What were the perceived benefits and problems brought about by offshore outsourcing? 

3. What were the perceived causes of the problems and how should the problems be addressed? 
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Methods 

Setting 

The study was conducted in the corporate office of one of the world’s largest package delivery companies 
and leading global providers of specialized transportation and logistics services. About 20 full-time employees and 
instructional design interns worked in the Corporate Training and Development Group. The Group was mainly 
responsible for the analysis, design, development, management, and evaluation phases of the training projects within 
the company. Both authors had experience working as instructional design interns with the Group. 

Participants 

Convenience sampling technique was adopted to choose two participants in the study. Both were directly 
involved in the offshore outsourcing. Cory (pseudonym) was in his mid 50s and was a staff manager at the Group 
while he was interviewed. He oversaw all phases of the training projects within the Group. He had more than 20 
years of working experience at the company and started his career as a package loader while in college. Henry 
(pseudonym) was in his mid 30s and was a project manager in the Group. Fresh from college, he started his career in 
the company as a computer programmer. After working 10 years in its information technology center, Henry was 
promoted and relocated to the corporate office. 

Data Collection 

The first author conducted a one-time interview with each participant by following an interview protocol 
created to guide the interview process. A tape recorder was used to record the interviews. The length of each 
interview was around 40 minutes. During the interview, the first author asked the participants to share their personal 
experiences of offshore outsourcing at the company, especially while they were working with the Corporate 
Training and Development Group. The first author also asked the interviewees to share their perceptions of various 
related topics including advantages and disadvantages of the offshore outsourcing, issues encountered during the 
offshore outsourcing, the causes of the issues, knowledge management, the Group’s core competencies, and learning 
curve related to offshore outsourcing. 

Data Analysis 

Miles and Huberman's (1994) data analysis procedures guided data analysis. First, we coded the transcripts 
into conceptual chunks and grouped the chunks into categories. Then we made sense of the relationship among the 
categories. And lastly, we wrote conclusions in the way of case narratives. 

Case Narratives 

Cory 

As a staff manager at the Corporate Training and Development Group, Cory first heard about the benefits 
offshore outsourcing could bring to American companies while he was participating in a professional development 
program in Washington D.C. in 1999. He believed that it was eye-opening experience. He stated, “After I saw the 
examples, I was amazed to see that it’s more efficient and less expensive to get work done. Plus, we can get more 
work done at a short period of time. And the quality was very good.” From then on, he began to understand that a 
global company like his has to look to offshore outsourcing to realize the multi-faceted benefits. 

Cory had taken a cautious and studied approach to offshore outsourcing. He was aware of the argument that 
offshore outsourcing could bring detriment to American economy by shipping off jobs overseas. After rigorous 
research, he changed his perspective. He believed that offshore outsourcing is a business function that “identifies 
some of the tasks that took us an enormous amount of time and cost us a lot of money, which we could outsource to 
people overseas who are more efficient and effective”. According to Cory, the Group’s core competencies reside in 
its understanding of what kind of training a person, a group, or a company needs and how to provide them. In other 
words, the analysis and design phases of instructional design and development is the core competencies for the 
professionals in training and development community. With the support from the top management, he started the 
initiative of offshore outsourcing. He outsourced some facets of instructional development including multimedia 
development, graphics, and computer programming to offshore companies such as the Tata Group, one of India’s 
largest business conglomerates. Cory did not believe that those facets are the core competencies of the training and 
development group, and he deemed it a strategic mistake to offshore outsource everything related to training and 
development. Cory quickly discovered that there were not well-prescribed guidelines on offshore outsourcing in 
training and development. He adopted a learning-by-doing strategy in his engagement in offshore outsourcing. He 
stated, “We just learn by going through the process, by going to talk to the Legal department in the company, by 
going and benchmarking with other companies in the consortium where we talked about offshore, by reading 
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research and magazine articles. It is a very informal process, driven by the desire to do it.” Ever since then, he had 
played multiple roles in this initiative including liaison, supervisor, champion, negotiator, and collaborator. Cory 
noted: 

My role has basically been to oversee the whole process, and make sure that when we sold this to 
other group or when their questions raised, I will be able to diffuse or neutralize some of their 
concerns. It’s also to encourage my team to continue this relationship top leverage this 
relationship, and to look for opportunities with other people’s projects to take them beyond they 
have never thought they can go. And at times, I sit down with the vendors. This is just part of my 
work. It is just part of life. Everything does not go perfectly. So when it does not, we sit down 
with them, and we say, “you know, this thing did not go, we were disappointed in what you did 
here.” And we will discuss those. ..And that’s the part of building the relationship forward. 

Cory perceived that offshore outsourcing increased the competitive advantage of the Group by 
strengthening its core competencies - development of training ideas and visions, analysis, design, storyboards 
creation, and identification of the path of learning, to name a few. Meanwhile, the outsourcing partners solidify their 
construction expertise in executing the training projects development. Offshore outsourcing had brought some 
unprecedented benefits to the Group, as Cory put it, “[offshore outsourcing] raises the bar of quality and 
productivity; it enhances our vision of what could be done and helps enable us to dream even bigger dreams.” 

Some issues such as language barriers and social cultural barriers came into play during the offshore 
outsourcing relationship. Eor example, Cory was surprised by some overseas outsourcing companies’ intention to 
come in and sell more before they have a chance to impress him. However, Cory regarded those barriers only as 
minor annoyances. Even though Cory admitted that the outsourcing partners did build their knowledge base on 
instructional design and development during the partnership, he was not concerned with the potential loss of the core 
competencies of the Group. Cory perceived that instructional design is a “very manual, intensive process” which 
offshore outsourcing partners would not be able to carry out on their own, as he noted. 

The things that they executed for us were not our core competencies, nor do I want them to 
become core competencies. What we hire these people for were things that we could not do in our 
Group, or did not make sense for me to hire people to do those in our Group. Basically it is our 
responsibility to, from the higher level, to chart the path of learning. These companies merely are 
our means to that end. So, we are not losing, in my opinion, any core competencies of any 
significance that will be detrimental to our Group. 

Cory perceived that the offshore outsourcing had exceeded all of his expectations. Eor example, the 
outsourcing companies had been reasonably easy to work with and they produced high quality work. Cory claimed 
that the partnership with the outsourcing companies had greatly enhanced the productivity in the Group. To 
summarize his experience with offshore outsourcing, Cory elaborated: 

We are the engineers. We are the ones who do the analysis and the design, and draw the 
storyboards. We give those to a builder. And that’s what I consider the relationship. In this 
relationship, they almost learn more from us in the business simulations that we did. And we learn 
from them. And the things that they know they did and could execute were not our core 
competencies, nor do I want them to become core competencies. What we hire these people for 
were things that we could not do in our group, or did not make sense for me to hire people to do 
those in our Group. Basically it is our responsibility to, from the highest level, to chart the path of 
learning. These companies merely are our means to that end. So, we are not losing, in my opinion, 
any core competencies of any significance that will be detrimental to our Group. 



Henry 

In early 2000s, Henry read a journal article on information technology outsourcing. That was the first time 
he learned about offshore outsourcing. His experienced offshore outsourcing first-hand after he started to work with 
the Corporate Training and Development Group. He was assigned to play the role of a liaison for training several 
projects development in India. At the time of the interview, he had been with the Group for about 2 years. Henry 
was in charge of the technological aspects of the training and development within the Group, and served as a 
technological liaison between the Group and information technology department, software vendors, and outsourcing 
companies. He explained his role this way, “you have a vendor who is very versed at training and you have a 
customer within the company who has a need, but does not know the training lingo. . .The role I have played is how I 
can bridge that language gap. . .” 

Like Cory, Henry perceived that the Group’s core competencies were its capability in analysis and design, 
and the Group only outsourced the construction phase of training and development projects to the overseas partners: 
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Our core competencies are that we are responsible for the performance of the workers. . .being able 
to look at how they do their job, what equipment they use, how we can change or increase their 
knowledge, or change their environment, or bring in some new technology, that will cause their 
performance to increase, therefore the company will do better... you cannot outsource that, you 
cannot outsource that analysis and design, that looking at how the company works... I think that’s 
what an internal group does, A vendor signs a contract to do a job, which means they are in that 
one instance. They don’t know the history; they don’t know the future and are not there for 
that... We are definitely outsourcing, that competency of being able to build training, and whatever 
technology that’s getting outsourced. 

Unlike Cory, Henry attributed the Group’s decision of offshore outsourcing to the lack of human resources in the 
Group - “by industry standards, we are undermanned.” The strategy of offshore outsourcing was nothing but to “fill 
in that gap.” Henry preferred a cautious approach to offshore outsourcing which should start with things that are not 
critical to the company. His ideal offshore outsourcing partners are the ones “who want the vested interest, who 
speak of the vested interest without me requiring of that. They want to be a business partner. They don’t want to be 
vendor. That way, they see my success as their success.” 

Henry was reserved about his perceptions about the benefits/advantages brought about by offshore 
outsourcing. Henry admitted that it was cheaper and more efficient to complete training projects overseas, and the 
quality was on par. Unlike Cory who depicted a mostly rosy picture of offshore outsourcing, Henry shared his 
several concerns. First, Henry expressed his concern on issues related to language barriers and corporate culture 
barriers. Unlike Cory, Henry did not restrict the language barriers to English, but more with language used within 
the company- “it is all those acronyms that we have within xxx (note: the company name)”. And he felt the urgent 
need to break the language barriers. He stated, “For vendors, no matter where they are out, in India or in China, the 
language will be foreign to them.” Moreover, the company he worked for has rich and unique corporate culture. 
Although he was a 10- veteran in the information technology branch of the company, when he relocated to the 
corporate headquarter, he encountered cultural shock. He believed that the difficulties experienced by the 
outsourcing partners were much more significant. Second, as a technical veteran, Henry was concerned with the loss 
of technical know-how. He used an analogy to illustrate such concern: 

It is like buying a car. You will never get enough knowledge to be able to look under the hood. 

You are always relying on the mechanics. And that’s fine. That’s what mechanics are good at. Buy 
you should at least be able to look under the hood and know what some of the stuffs are, or at least 
be comfortable to ask the mechanics questions “I have an issue with this”. That way, they are not 
taking advantage of you because you just don’t know. 

He was concerned that, in the near future, such immediate benefits as low cost might disappear. But at that time, 

“we lost all that competency and knowledge to do it. What are we going to do?” The reality will be more poignant 
since the outsourcing partners does not share a vested interest in the company, just as he put it - “They are the 
people that we should have in house that fill in the gaps. . .they have all that knowledge to build all those things and 
we simply don’t have. At some point, if they are going a different direction and they go somewhere else, there goes 
all those knowledge.” Third, Henry expressed his concern about the trust issue and strategic partnership with the 
outsourcing partners. He realized that it is difficult to create and maintain a strategic long-term outsourcing 
partnership. If the outsourcing partners are not interested in building a sustained strategic partnership, then they 
don’t have a vested interest in your company’s growth and survival. He thus favors to maintain the in-house 
construction competencies, because he believes that the employees’ vested interest in the company’s growth and 
survival is exactly the reason “why we hire employees. They become a part of a family, and we treat it as a culture 
and as a family.” He further elaborated the reasons for maintaining in-house competencies: 

One big issue is when you outsource something, it saves you money right now. It could be done 
right now really quickly. What happens a year or two years later when that has to be updated? Is 
that vendor A still around? Or are they still committed to you and your company as they were 
back then? When you have something done in house, I am committed to doing it because I am 
paid by XXX (note: the company name). There is a vested interest in me completing that project to 
the best of my ability... But when a vendor does a job, they get paid by XXX. But do they, in 
essence, have that same vested interest! Or will they be concerned about XXX as a company 
doing well, so that they can also get more work. I am not quite sure they have that. I don’t believe 
they have that same vested interest. 
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Conclusion 

Overall, both Cory and Henry perceived that the outsourcing companies had exceeded all of their 
expectations, including cost effectiveness, work efficiency, work quality, and timeliness of the project completion. 
As far as the issues related to offshore outsourcing, both of them cited language barrier, corporate culture difference 
and social cultural differences, but viewed them as minor issues. Both Cory and Henry perceived that the group’s 
core competencies reside in its ability to conduct high level analysis and instructional design; the group did not lose 
the core competencies in this outsourcing relationship. Although they thought that offshore outsourcing was a win- 
win situation for both parties, they held different perspectives of the long-term effects of offshore outsourcing. Cory 
perceived that his group did not experience any knowledge/expertise loss during the relationship; on the contrary, 
his group gained development expertise from the outsourcing company. Although he acknowledged that the 
outsourcing company did learn some instructional design strategies from his group, he did not feel threatened that 
the outsourcing company would be able to quickly build up its expertise or knowledge base in instructional design to 
compete with the core competencies his group held. In his view, the outsourcing company was merely their means 
to an end. Henry was less optimistic. He expressed his concerns on whether his company will maintain a long-term 
strategic partnership with the outsourcing company so that the outsourcing company has vested interest in his 
company. He experienced the loss of his expertise in computer programming and software development, because of 
his change of role from a programmer to a manager. He was worried about the long-term negative effects of the 
over-reliance on the outsourcing company for its development expertise. If the outsourced company stops working 
with his company, it will be difficult for the group to update the computer-based training projects such as 
simulations and games. Given this, he suggested that the group should be cautious in dealing with offshore 
outsourcing and take small steps to evaluate if that is the appropriate strategy. 

The results of the paper reveal that, the professionals in training and development who are involved in 
offshore outsourcing need to further develop their competencies in interpersonal foundation and 
business/management foundation including (a) building trust with outsourcing partners to develop a healthy long- 
term strategic partnership, (b) applying their business acumen in managing the partnership, and (c) thinking 
strategically how to avoid the loss of knowledge during the outsourcing partnership. The results reported in this 
study were worthy of continued investigation. With this study, the authors hope to add to the literature on how 
professionals in the field of training and development should critically and strategically deal with offshore 
outsourcing. 
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Abstract 

Current approaches to workplace learning emphasize designing communities of practice that are intended 
to support both formal and informal knowledge acquisition. This paper presents the design and research of a 
knowledge-based community of practice for Subaru, Inc., based on principles outlined by Scardamalia (2002) and 
Zhang et al (2007). The purpose of this study was to investigate the extent to which participants’ interactions in the 
online community showed evidence of individual and collective knowledge building. We found evidence of 
knowledge building within online discussions in these areas: interactions around improvement of ideas, connection 
to workplace knowledge and practices, and building on and/or adopting the ideas of others. We also found 
significant gains in scores on an assessment of workplace customer service, after participation in the online 
community of practice. 

Introduction 

Today’s organizations place great importance on the creation and reification of knowledge (Penuel & 
Cohen, 2002). Instructional technology has enabled corporations to effectively manage explicit job-related 
knowledge, through training, manuals, and other forms of knowledge sharing that are often available electronically. 
Cook and Brown (1999) suggest that explicit knowledge is regarded as formalized and can be documented. Explicit 
knowledge is easily articulated and takes the form of documents, websites, customer relationships management 
(CRM) databases, and manuals that can be shared, and transferred to others. Corporations construct explicit forms of 
knowledge such as formal procedures, manuals, training and job descriptions. Traditional training approaches are 
pedagogically aligned with supporting acquisition and sharing of explicit knowledge. 

However, organizations are less effective at managing and facilitating sharing of more informal, tacit 
knowledge. Tacit knowledge is the result of knowing and utilizing explicit knowledge through practice. Tacit 
knowledge becomes manifest in work practices and is described as ‘know how’ transferred by story telling, 
conversation, experience, and narrative (Gray, 2004). Tacit knowledge is acquired largely from experience while 
performing everyday activities, but typically without conscious awareness of what is being learned (Schon, 1983). 
Organizations benefit from the conversion of tacit knowledge into explicit knowledge. Orr’s (1999) research study 
of Xerox service technicians illustrates that while manuals are helpful to document knowledge for the organization, 
explicit knowledge is dependent upon tacit knowledge to be truly effective. Indeed, an interaction between explicit 
and tacit knowledge exists (Sternberg & Horvath, 1999), such as knowing if a particular ‘rule of thumb’ applies in a 
particular workplace situation. 

Current approaches to workplace learning emphasize designing communities of practice that are intended 
to support both formal and informal knowledge acquisition (Gray, 2004). Such communities aim to support not only 
the acquisition of explicit knowledge and skills, but also the “informal and social aspects of creating and sharing 
knowledge’’ (Gray, p. 23). Social anthropologist, Jean Lave and social learning theorist Etienne Wenger introduced 
the term community of practice (CoP) to describe a group of individuals who share similar interests and through 
interaction and activities collectively develop new practices and knowledge. CoP’s explain how individuals learn 
through everyday social practices and is often defined as “a group of people who share an interest in a domain of 
human endeavor and engage in a process of collective learning that creates bonds between them’’ (Wenger, 2001, p. 
1). A CoP transcends explicit knowledge and skill associated with an activity or task, and members engage in 
interaction and collaboration over a period of time. (Lave & Wenger 1991). 

Recent efforts have focused on using technology to intentionally design learning environments for 
communities of practice that emphasize informal learning, knowledge sharing and knowledge building (Hoadley & 
Kilner, 2003; Scardamalia, 2002; Zhang, et al., 2007). Through participation in a knowledge-building community, 
individuals learn both the skills and culture of their practice. Knowledge-building communities are defined as social 
activity systems that target “communal knowledge creation supported by collective and sustained idea 
improvement’’ (Zhang, et al., p. 1 19). Members in a knowledge-building community have a dual role — to 
collectively and individually take the initiative for knowledge advancement. 
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Knowledge building has typically been studied in school-based environments (Scardamalia et ah, 1989; 
Zhang et ah, 2007). A knowledge-building community supports collective knowledge building and discourse, with 
an emphasis on interactions based on real ideas or authentic problems and that result in idea improvement (Zhang et 
ah). Scardamalia (2002) forwarded 12 knowledge-building principles to inform design. Our study examines three of 
these principles that are most tied to the work presented here and further informed by Zhang et al: (a) idea 
improvement, or advancing one’s ideas by responding to problems, questioning, or elaborating/contributing new 
ideas; (b) real ideas/authentic problems, or advancing ideas and work-related practices through sharing of first-hand 
experiences; and (c) community knowledge, shown as the interplay between individual knowledge advancement and 
community knowledge as a whole. 

Research on knowledge-building communities has shown improvements in children’s achievement and 
self-regulatory strategies (Zhang et al., 2007). The concept of knowledge building within a community model has 
clear ties to goals for fostering workplace communities of practice. However, little work has been done to extend 
Scardamalia’ s (2002) knowledge-building framework to corporate contexts. This paper presents the design and 
research of a knowledge-based community of practice, based on principles outlined by Scardamalia and Zhang et al. 
The purpose of this study was to investigate the extent to which participants’ interactions in the online community 
showed evidence of individual and collective knowledge building. We sought to understand the nature of knowledge 
building in the online community, based on content analyses of online discussions and indicators of connections to 
everyday job-related practices. To accomplish these goals, we investigated the following research questions: 

1 . How do participants engage in improvement of their own and others’ ideas during online dialog around 
best practices and solutions to problems? 

2. How do participants make references to real ideas and authentic experiences during online dialog? 

3. How do individual contributions lead to the development of the community’s ideas more broadly? 

4. Is participation in a knowledge-building community of practice associated with improved job performance? 

Method 



Participants and Context 

Thirty-eight employees from Subaru dealerships located across the country participated in the study. The 
eight- week training initiative was part of Subaru’ s customer service training curriculum. A knowledge -building 
CoP was created using a course management system (Moodle) for participants to access eight weekly topics. 

Online discussions focusing on specific content, and question prompts enabled participants to share individual 
experiences and best practices with the community. Building Service Excellence is the cornerstone customer service 
course and required of all Subaru dealership personnel. The course material introduces Service Advisors to basic 
customer service skills that are essential to maintain and increase customer loyalty. 

The content was converted from a classroom-based course and customized to include industry-specific case 
studies, best practice development, and online discussion. The course was delivered over an eight-week period with 
seven weekly topics. The learning environment was accessible 24/7 for the duration of the course and the learners 
were able to complete the course from work or home. Building Service Excellence courses contain seven topics: 1) 
Personal and Practical Needs 2) Steps to Service 3) Key Principles 4) Taking the HEAT 5) Walkers and Talkers and 
6) Recovery and 7) Action Planning. These modules were broken out into 8 weeks, and utilized a combination of 
the following strategies: (a) instructional materials to deliver course content, including presentation slides with 
audio narration, text, and electronic documents; (b) scenario-based problems requiring participants to generate a 
position or solution; (c) sharing of ideas and best practices through story telling and making connections between 
content and practice; (d) “action planning’’, or planning a series of steps or actions to solve a known performance 
problem, based on customized report-generated data. 

Activities and discussions were designed to reflect three primary knowledge-building principles 
(Scardamalia, 2002): (a) idea improvement; (b) real ideas/authentic practices; and (c) community knowledge. These 
specific knowledge building principles guided discussions, activities and best practice development by transforming 
knowing in action to knowledge in action. Activities required participants to share work-related experiences and 
ideas through storytelling to solve on-the-job problems related to customer service issues and to develop best 
practices that will increase customer loyalty. The goal of the community of practice was to foster an individual 
member to share experiences with the community. The stories that are shared become the community’s collective 
knowledge and those community members have the opportunity to critically reflect, analyze, and form new 
perspectives that impacts individual on-the-job performance. 
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Data Sources 



The main data sources used to address research questions 1-3 included online discussion posts. Transcripts 
of all online postings were created and coded using Zhang et al’s (2007) and Scardamalia’s (2002) frameworks as a 
guide. A team of researchers defined the coding scheme based on theoretical definitions for idea improvement, real 
ideas/authentic problems, and community knowledge. A minimum of two researchers coded the data separately, 
meeting an overall 81% interrater agreement for all codings. Where disagreements existed, an adjudicated rating 
was used. Research question 4 used job-performance scores from the Owner Loyalty Program (OLP) report. These 
reports compile customer survey scores for a given dealership and are reported quarterly. Quarterly reports from 
before participation in the CoP (Q3) and after (Q4) were collected and compared to determine if gains occurred. 
More detail on each data source and analysis technique are detailed as follows: 

Idea Improvement. Online postings were coded to determine patterns of idea improvement. Based on 
Zhang et. al (2007), we defined idea improvement by examining three elements: a) responding to a problem, idea or 
comment in a way that advances or elaborates the original idea (e.g. “1 agree with you on every point. It is essential 
to keep the customer in the communication loop — even when you don't have anything to report”); b) raising 
question(s) about a problem, idea or comment for peers to address, including clarification questions, elaboration 
questions, what-if questions, and challenge questions (e.g. “Wo(u)ld it be possible for you to send a sample copy of 
the survey home with them?”); c) contributing new and original idea(s) within a discussion unit, including starting a 
new thread about original idea(s), taking a thread in a new direction, and raising question(s) that lead to a new 
direction (e.g. “My other two manufacturers provide a folder which includes the PDl check list and the Delivery 
checklist which can be completed and kept in the deal jacket for future reference. Does Subaru provide something 
similar?”). 

Real Ideas/Authentic problems. Based on Scardamalia’s knowledge building principles, we categorized 
participants’ postings as reflecting real ideas, authentic problems to the extent that they expressed prior, on-the-job 
experiences to frame a discussion, problem, or solution. These postings were typically expressed as solutions to real 
problems or as real stories about experiences (e.g. “1 have done comment cards in the past and second party 
phone calls and also we tried a computer generated phone call system which 1 did not like the service 
provided”). Scardamalia (2002) proposed that knowledge building is stimulated when real ideas and problems 
interact, in contrast to interactions around abstract or decontextualized concepts. 

Community Learning. Online postings were analyzed to determine whether individual participant ideas 
contributed to the work and ideas of others in the community more broadly. That is, did participant’s written posts 
reflect reading of, and integration of, others’ posted ideas? Participant postings were coded according to whether 
they represented a response that suggest either building upon ideas expressed by others or an expressed intention to 
adopt an idea from another in their own work practices. Building upon ideas was defined as taking an idea or 
comment contributed by another participant and building upon it to create a new idea or take the idea in a new 
direction. Adopting ideas was defined as a participant’s expressed intent to take an idea advanced by a colleague and 
apply it to their work practices. 

Work-related Practices. Improvements in job performance were assessed by comparing Owner Loyalty 
Program (OLP) scores before and after participating in the knowledge-building CoP (period of Q3 and Q4 2007). 
Complete OLP data were only available for 22 of the 38 participants; thus, only those 22 participants were included 
in the analyses. Scores used were the Subaru Owner Loyalty Index (SOLI) and Subaru Owner Loyalty Index Plus 
Recovery (SOLl-t) measures (metrics tied to the concepts and discussions in the Building Service Excellence 
course). OLP data measures customer satisfaction with their purchasing and service experiences with data collected 
on a quarterly basis. To evaluate a dealership’s loyalty and customer service we used SOLI and SOLl-t scores. 

SOLI scores are measures of owner loyalty, computed through responses to customer surveys. Customer loyalty 
strategies were discussed throughout the CoP. Customers are considered “loyal” if they check the maximum 
satisfaction box for three specific questions on the survey. If any of the customer responses for these questions are 
below the maximum response, the customer is considered “disloyal.” The dealership’s SOLI score is the percentage 
of loyal customers. 

The BSE program was delivered in Q3 2007. If participation was associated with a positive impact on 
Subaru customers’ experiences, then SOLI and SOLl-t scores for dealerships participating in the program would be 
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expected to increase from Q3 2007 to Q4 2007, while the OLP score standard deviations would decline as 
individuals adopted more homogeneous service practices. A paired-sample t-test was conducted to test for gains in 
SOLI and SOLI-r performance between Q3 and Q4. 

Procedures 

The CoP was designed and developed by two instructional designers, and all content materials and 
discussion forums were uploaded to, and delivered by, Moodle. A team of instructional designers and educational 
researchers (including one from Subaru, Inc.) vetted the knowledge building environment’s construct validity, based 
on examining a design document that paired instructional strategies for each lesson with associated knowledge 
building principles. Participants then enrolled in the course, and completed the informed consent. The participants 
proceeded through the online environment for eight weeks. Each week, participants were asked to complete the 
required readings and activities and post to the assigned discussion forum. The “Best Practices” forum was 
introduced to the participants in the second week of the course, and was available for posting throughout the course. 
The course facilitator (who was also an instructional designer and co-researcher) responded to discussions and kept 
the course proceeding on pace. At the end of the course, participants were asked to complete a survey about their 
perceptions of their experience. 



Results and Discussion 



The results of the study are presented in the following sections according to each research question. 



Idea Improvement 

A key element of knowledge building involves building off of the ideas advanced by the community 
(Zhang et ah, 2007). Member ideas are stored collectively in discussion threads or electronic documents that can be 
used as a foundation for idea improvement. In this study, participants engaged in online discussions weekly, based 
on the assigned tasks. Adapted from Zhang et. al (2007), we defined three elements of idea improvement; a) 
advancing ideas by responding to a problem, idea or comment (Type a); b) raising question(s) for peers to address 
(Type b); c) contributing original idea(s) by starting a new post or leading a discussion in a new direction (Type c). 

As the tasks, scenarios and discussion topics differed across weeks, the length of each week’s discussion, 
and hence frequency of idea improvement postings, varied. Figure I presents the total percentage of three types of 
elements, while Table I reports the frequent counts for each week. Participants engaged in discussions frequently by 
addressing factual or explanatory problems (Type a) as well as contributing personal ideas (Type c). They were less 
active in raising questions for peers to clarify or elaborate on an original post (Type b). The difference of frequency 
counts between Type a and Type c elements is within I to 4 posts on a weekly basis, except for the second task in 
week 2. For week 2 (task 2), there were forty seven Type a elements while only 2 Type c elements (see Table I). 



Figure 1 Percentage of three types of Idea Improvement elements 




A 



Type c- original 
contribution 
32 % 





■ Type a- 
responding 

■ Type b- 
questioning 

■ Type c- original 
contribution 
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Table 1 Number of three types of Idea Improvement elements in the postings 





Number of Type a 
responses (Responding) 


Number of Type b 
responses (questioning) 


Number of Type c 
responses (contributing 
original ideas) 


Week 2 (Task 1) 


11 


2 


12 


Week 2 (Task 2) 


47 


2 


2 


Week 3 


9 


2 


13 


Week 4 


9 


2 


7 


Weeks 


9 


3 


6 


Week 6 


4 


1 


4 


Week 7 


9 


2 


5 


Weeks 


6 


2 


3 


Best Practice Forum 


28 


11 


25 



Although the flow of idea development varied from unit to unit, we found two different patterns of idea 
improvement flow: (a) from contributing original ideas (type c) to responding to problems, ideas, or comments (type 
a); and (b) from responding to problems, ideas, or comments (type a) to contributing original ideas (type c). The 
patterns of idea improvement flow took place within each thread. Week 7 showed some examples of the first pattern 
of idea improvement flow. According to the unit task, each participant presented an action plan including his or her 
own ideas and solutions to existing problems, followed by comments and suggestions from peers. An example of the 
second flow can be found in week 4, when participants first responded to the given questions “whether the customer 
is a walker or talker?” and “Why?” based on the scenario presented. Rooted on their own or peers’ answers, some of 
the participants advanced their ideas by relating this scenario to their work experience and stated how they would 
solve the problems. In both instances, the nature of the task defined the interaction pattern that emerged. 

According to Zhang et. al (2007), deeper understanding and knowledge building occur when participants 
generate questions. However, in our study, this was rarely the case. Participants more commonly responded to the 
given scenarios or commented on peers’ responses, and generated original ideas concerning a problem. In contrast to 
Zhang et al. (2007), this finding could be a function of the nature of workplace knowledge-building, where perhaps 
less emphasis is placed on questions about explicit knowledge and instead more emphasis on articulating real ideas 
and connecting to others’ real experiences. 

To sum up, the above findings suggest that in our CoP, participants showed evidence of engaging in dialog 
that contained elements of idea improvement. Different from Zhang et. al (2007), our participants did not raise 
questions frequently to generate deeper understanding of explicit knowledge; instead, they tended to build on their 
everyday experiences and look for solutions to future problems. 

Real Ideas, Authentic Problems 

Scardamalia’s (2002) principle of “real ideas, authentic problems” is a necessary element in the knowledge 
building. Community members interact about concrete, real situations that impact their daily lives by creating and 
reflecting on ideas collectively using empirical, conceptual artifacts to frame discussions (e.g. “I think you are on 
the right track, giving them a check list to follow sets a process for delivery. Perhaps have them sign off on 
the list and return to you for tracking purposes”). Adapting Zhang et al’s (2007) knowledge building principles, 
we defined real ideas, authentic problems as responses that expressed prior, on-the-job experiences to frame a 
discussion, problem, or solution. Statements that reflected real ideas/authentic problems were frequently 
identified in each week’s discussion and resulted from the story telling process. An example statement includes 
the following: “When someone calls in for a service and we know it will take more that a couple hours we ask them 
if they need a loaner. Then we schedule their appointment according to the availability of a vehicle. We have been 
using this process for the last several years and I believe it has helped us to get many more compliments and 
additional service work.”). 

As Table 2 illustrates, we found that the use of real ideas and authentic problems occurred more 
frequently in the early weeks, but that the highest frequency of such responses were found in the “best 
practices forum”, which was referenced throughout the course. As the discussion activities went on, the 
frequency counts of the real ideas and authentic problems decreased 50 percent in the fourth week, and it 
returned to its peak again in the fifth week. After week 5, the number of real ideas and authentic problems 
decreased, as participants seemed to focus on completion of final course activities. The nature of the 
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discussions in weeks 2,3 and 5 entailed responding to small case scenarios and/or articles by making 
connections to challenging customer service situations encountered in the workplace. 



Table 2 Number of Real Ideas/Authentic Problems in the postings 



Weekly Discussion 


Real ideas/authentic problems 
Frequency Counts 


Week 2 


14 


Week 3 


16 


Week 4 


7 


Weeks 


15 


Week 6 


7 


Week? 


9 


Weeks 


5 


Best Practice Forum 


42 



Community Knowledge 

Community knowledge was identified as those responses that reflected building on or adopting ideas 
advanced by others in the community. Across the eight- week course, we found a total of 13 building upon idea 
statements and 1 1 adopting idea statements. Eight of these statements, four in each category, were found in the best 
practices forum, which was not unexpected given the nature of the best practices forum as a place to share and 
discuss best practices honed in the dealerships. In such instances, participants would write about how they dealt 
with a specific work-related problem that they felt was significant enough to share with the group broadly as a “best 
practice.” 

For postings that reflected “building upon” statements, participants typically expressed agreement or 
disagreement with a real-world practice posted by a member of the community, and added to, elaborated, or 
qualified, the initial idea with more information relevant to the participant’s own experiences or situations. For 
instance, one participant mentioned; “If my service manger came to me with this issue, I would suggest to him that 
the next time the customer came in to let me, as the sales manager, discuss the survey with them. It could be 
possible that if the explanation came from a different individual they would understand.” Likewise, one participant 
posted the following “building upon” sentence: “. . .1 also agree. Great tactic! For instance, from the customer side, 
I know I feel more comfortable when a sales/service person repeats what I am trying to say.” In this case, the 
discussant was building upon a posted suggestion that colleagues should try reducing the physical space between 
themselves and a customer when a customer was upset with a service-related issue. 

In terms of responses that represented “adopting” ideas, we found that specific concrete recommendations 
were the most likely to be “adopted” by others. An example of an adopting ideas response included this response to 
a suggestion that was advanced for dealing with water leaks by checking the sealant on the car’s drain tube block 
(e.g., “We rarely get water leaks in Subaru but if the next has a sunroof we will certainly check this.”). In another 
example, one participant wrote the following: “[I] Like the idea of the card left in the car. Will have to give that 
some serious thought.” 

The building on and/or adopting of experiences articulated in the community holds potential as a means for 
sharing of tacit knowledge, as illustrated in the following comments: 

I had been having problems getting people to respond to surveys for warranty work. And, I was trying 
to figure out the best way to get more people to respond and not just the people that were not happy. 
And, I got some good ideas about mailings in [the discussion forum]. 

... in Subaru, when they used to send a car in, they would send a delivery checklist. . . so when our 
salesmen would deliver the car, they would go through this checklist and check off everything, and 
have the customer sign it . . .and then it would be kept in the deal jacket so if there were any issues we 
could always go back to that. . . And then this [participant from] California said - ‘Hey, you know, that 
was a real problem, but then we developed this, and we use this now. . .’ and he actually posted it to the 
site. . . and then I pulled it off, changed it for our dealership. . .and we are using that now. . . 

In sum, although comparably low in occurrence to other knowledge-building activities, building/adopting responses 
were frequently associated with concrete on-the-job problems that were of broad interest across the community. 
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Furthermore, some anecdotal evidence shows that participants actually used some of the ideas advanced by the 
community to solve existing workplace problems. 

Work-related Practice 

Improvements in job performance were assessed by comparing Owner Loyalty Program (OLP) scores 
(SOLI and SOLI+) before and after participating in the knowledge -building CoP (period of Q3 and Q4 2007). Table 
3 shows that the mean SOLI and SOLIh- scores increased from Q3 to Q4, and their respective standard deviations 
declined from Q3 to Q4. 



Table 3: Descriptive Statistics for SOLI and SOLIh- Scores for Q3 and Q4 





N 


Minimum 


Maximum 


Mean 


Std. Deviation 


Q3 SoliH- 


22 


66.7 


100.0 


89.318 


8.0399 


Q3 Soli 


22 


33.0 


100.0 


82.005 


15.2352 


Q4 Soli H- 


22 


80.0 


100.0 


95.414 


5.6941 


Q4 Soli 


22 


80.0 


100.0 


92.882 


7.4241 


Valid N (listwise) 


22 











In order to evaluate the statistical significance of this improvement, a matched-pair t-test was performed. Table 4 
reports the results of these tests and shows that participants improved their OLP scores by a statistically-significant 
margin from Q3 to Q4 for both SOLI and SOLIh- Scores (f(21)=-4.565, p<.001; f(21)=-3.594, /?<.01, respectively. 



Table 4: Paired Samples T-test for SOLI and SOLIh- 







Paired Differences 




















95% Confidence 


















Std. 

Error 


Interval of the 


















Difference 














Mean 


Std. Dev 


Mean 


Lower 


Upper 


t 


df 




Sig. (2-tailed) 


Pair 1 


Q3 SoliH- 
Q4 SoliH- 


-6.0955 


6.2625 


1.3352 


-8.8721 


-3.3188 


-4.565 


21 




.000 


Pair 2 


Q3 Soli 
Q4 Soli 


-10.8773 


14.1938 


3.0261 


-17.1705 


-4.5841 


-3.594 


21 




.002 



While these data are not conclusive as to direct impact, they nonetheless support the possibility that participation in 
the BSE CoP was associated with individual performance gains of those who participated. 

Closing 

In summary, participants were able to share individual knowledge to generate collective idea improvement 
and share real ideas, authentic problems based on the discussion post analysis. Knowledge-building activities are 
not an inherent characteristic of knowledge building CoPs and require structure for individual members to achieve 
deep understanding and collective advancement of knowledge. One implication for the design of knowledge- 
building CoPs is how to better support the reification of an individual’s tacit knowledge into collective explicit 
knowledge. 
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Labels DO Matter! A Critique of AECT’s Redefinition of the Field 

Patrick Lowenthal 
Brent G. Wilson 
University of Colorado Denver 

AECT has recently (yet again!) redefined our field, reverting back to the use of the term educational 
technology. We believe this recent change is problematic for a number of reasons, but primarily because of 
the weak rationale offered for the change. This change affects how external audiences view our profession 
and is likely to confuse practitioners in corporate and higher-ed settings in particular. We offer a review of 
job postings, program titles, and listserv discussions to support our case. The labels we use to define 
ourselves ARE CRITICALLY important - and we hope to see a stronger case made for changes for our 
foundational definitions in the future. 



Introduction 

Students, academics, and working professionals struggle with some of the naming and labeling conventions 
used in our field. Most professionals in the field struggle with the cocktail party question - “So tell me - what is it 
that you do exactly?” The question requires some careful thought and a sentence or two in reply, since few people 
are really familiar with the names we use to describe ourselves. Ongoing exchanges on the Instructional Technology 
Eorum (ITEORUM) listserv and the Distance Education Online Symposium (DEOS) listserv are further evidence of 
this. Eor instance, in May 2007 a graduate student sent a question to the ITEORUM listserv asking: 

What's the most efficient/marketable/current name for folks who do what we ID folks 
do? . . . there are other graduate programs out there for Educational Technology, 

Instructional Technology, etc. I'm an instructional designer right now, but my boss asked 
me what I wanted on my new business card, and a quick search of the job boards shows 
all of the above titles. And from what I can read, all those programs produce graduates 
who do close to the same thing. 

In response, Clark Quinn responded: 

I've been bemoaning for years that we have a labeling problem. Instructional and 
Educational are both so limiting, implying as they do only formal learning solutions 
(which can lump you into either the 'school' group or worse, the 'training' group, which 
gets no respect). However, Learning Technology, while somewhat better (can include 
more forms of learning than just education or instruction) still sort of misses the 
performance support category. And then. Performance Technologist might mean pump 
performance, or financial performance, or... and I find 'human performance technologist' 
kind of weird (a sports trainer? a sexual therapist?). 

This year AECT published a new book entitled Educational Technology: A Definition with Commentary 
(Januszewski & Molenda, 2008). It is this redefinition - particularly its adoption of the term educational technology 
- that we respond to in this paper. 



Background 

Among the different professional organizations at the center or periphery of our field (e.g., AECT, ISTE, 
ISPI, ASTD, AACE, SALT), AECT historically has been the most influential in shaping and guiding our field. As a 
veteran organization dating back to the early audio-visual movement, AECT is the only group to systematically 
attempt to define the field over the years (Seels & Richey, 1994). In 1963, 1972, 1977, 1994, and now in 2008, 
AECT has published official definitions of the field meant to serve as a conceptual foundation for theory and 
practice. The focus of our paper is to critique one aspect of that redefinition: AECT’s decision to return to the use of 
the term educational technology. 

Both Seels and Richey (1994), authors of Instructional Technology: The definition of the field, and 
Januszewski and Molenda (2008), authors of Educational Technology: A Definition with Commentary, agree that the 
terms educational technology and instructional technology are often used interchangeably. Even so, some 
distinctions are commonly made (Gentry, 1995). Both insiders and outsiders to the field suggest that educational 
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technology suggests a greater focus on K12 issues and instructional technology a more generic reference to 
instructional settings of all types. These perceptions are sometimes entrenched - for example, one of us had an 
Associate Dean of Distance Learning tell him that she would not even consider hiring someone with a degree in 
Educational Technology because she believed they were not prepared the same way as graduates with a degree in 
Instructional Technology or Instructional Design. 

AECT’s definitions have followed an interesting route in their use of these two terms. In 1972 and 1977, 
AECT officially adopted the label educational technology, however, in 1994, AECT began officially adopting the 
label instructional technology, with the publication of Instructional Technology: The Definition of the Field (Seels 
& Richey, 1994). The authors devoted a six-paragraph section to justifying the decision to change the label from 
educational technology to instructional technology. Disappointingly, the following is about the extent to which 
Januszewski and Molenda (2008) specifically address the change: 

[T]his book presents a definition of the field of study and practice as “educational technology” or 
“instructional technology.” While recognizing that educational and instructional have different 
connotations, the authors intend that this definition encompass both terms. It could be argued that 
either term is broader and more inclusive in some sense, but the current definition and Terminology 
Committee chooses to focus on the sense in which education is the broader term. (p. ix) 

In a later chapter on implications for academic programs, Persichitte (2008) suggests: "[T]he important 
point is not whether the definition (or the program title) is educational technology or instructional technology or any 
other combination of relevant terms" (p. 332). 

In both cases the authors seem to be accepting a couple of tacit points: 

• One way, or perhaps the best way, to choose a label for our field and practice is by choosing the more 

general or broader label 

• Labels, or at least the label of educational technology or instructional technology, do not matter very much 

and that the changing of these labels does not warrant a full rationale 

Choosing a label though should be a bit more complicated than this and require more forethought. How a 
professional organization labels a field can have far-reaching consequences both for members within the 
organization as well as those outside of it. Thus, changing a label of a field should be more than an academic 
exercise of picking the more general of two terms (that are sometimes used interchangeably). 

This may be seen by the definition's authors an unfair characterization of their position - and we frankly 
hope this is the case. The problem is that a thorough justification for the change cannot be found in the book - 
leaving readers to piece together some kind of grounds for the decision. 

Labels Matter 

Of course, the choice of label for our field matters, even for words often used interchangeably. Before 
looking at some empirical evidence of how and when certain labels are used, we review below some reasons why 
labels are important. 

Market and Branding 

Connotations of words refer to shades of meaning that color or suggest association, but do not concretely 
change the referent. Connotations of labels have important impacts because people are drawn to certain names and 
repelled by others; hence the considerable investment in branding and promotion by marketing specialists. A brand 
is seen as a primary asset by an organization, reflecting years of investment in quality and promotion. Companies 
consider a change in a label only very reluctantly, understanding that the market may perceive a name change as a 
sign of trouble in the brand. 

Language and thought 

Language is intricately connected to thought and learning (Vygotsky, 1962, 1978). Just as language shapes 
thought and social practices, thought and social practices shape language. Language is the foundation of culture 
(Ong, 1982) and plays an important role in communities of practice such as ours (Wenger, 1999). The way we 
communicate reveals who we are and how we think. Language is never value neutral (Bourdieu, 1970). The 
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language that is used by an organization can shape how members of the organization think. We need to reflect on the 
language and the labels that we use and how it impacts our thinking and our field. Despite the early work by 
psychologists like Vygotsky (1962), and later work by linguists like Gee (1996) and cognitive scientists like Lakoff 
(1987; Lakoff & Johnson, 1980), we seem to forget — or not even acknowledge in the first place — how the language 
we use influences and structures thought. Moreover language, or more specifically discourse, is not merely the 
transference of ideas from one person or people to another; it is the very making of meaning and shaping of 
identities. 

Identity 



We create ourselves and are created through the language we use (Bruner, 1986, 1990, 2002; Gee, 2002, 
2003; Vygotsky, 1962). Thus, a change in language use can bring with it changes in identity. Theorists now agree 
that we each have multiple identities (Gee, 2003); identity is not a static unchanging entity but rather a dynamic 
entity that is influenced by the communities of practice we are a part of and their associated uses of language (Gee, 
2000). In fact, Wertsch (1991) has argued that language plays a crucial role in an individual’s inclusion within a 
specific environment or culture. While simply changing a label or adopting an older alternative, might seem like a 
minor change, there is reason to believe based on the literature on identity and language that a change like this could 
have a greater impact than some realize. Specifically, when the label or language that is changed is directly related 
to name of a field of practice. 



Perspectives from the Field 

While definitions of the field may be of primary interest to academics, many practitioners have an interest 
in how we label and talk about the field - especially if those definitions affect their jobs. To get a feel for how day- 
to-day practitioners talk about the field, we reviewed professional job announcements, the titles of academic 
programs, listservs, and book titles. 

Professional Job Announcements 

Job postings seemed the natural starting point. Interestingly, before we began analyzing the titles of job 
postings, we noticed that neither the Chronicle of Higher Education nor HigherEdJobs.com labeled jobs in our field 
educational technology . For instance, on the Chronicle's website, you must select one of the following options 

Instruction design 

Instruction development 

Instructional technology 

Instructional technology education 

Instructional technology/design 

Instructional technology/design (campus) 

Similarly, if you search on higheredjobs.com by type, our field is labeled as instructional technology and design. 
While we chose not to sample job postings from InsideHigherEd.com, we noted that they too do not have an option 
to search for jobs in the field of educational technology — instead they label our field as instructional technology/ 
distance education. 

We purposefully sampled job postings from national job boards that are known to publish vacant positions 
in our field. The following web sites were purposefully selected for this study because of their overall popularity 
(e.g., the Chronicle and HigherEdJobs.com both list more positions in our field than most other employment web 
sites combined); 

1 . Chronicle of Higher Education 

2. HigherEdJobs.com 

3. AECT 

4. University of Indiana’s Instructional Technology Job Board 

Six weeks of job postings were compiled from each employment website. The postings were copied and 
pasted into an excel spreadsheet. The data was then cleaned up. Finally, the data was compiled and analyzed. While 
the name of a field does not have to correspond to specific job titles, we did expect to find positions for such things 
as educational technologists as well as instructional technologists and finally instructional designers. We found 
though that of the 327 job postings we analyzed, only 9 of those positions had the label educational technology in 
the job title. On the other hand, we did find that label instructional technology was used 29 times or more than 3 
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times as often as educational technology. Interestingly, instructional design was used the most often; see Table 1 
for a complete list of what we found. 



Table 1 



Labels Used in Professional Job Announcements 



Source 


Total Job Postings 


Educational 

Technology 


Instructional 

Technology 


Instructional Design 


Indiana University I.T. 
Job Board 


52 


1 


9 


19 


AECT Job Board 


17 


0 


5 


1 


HigherEdJobs.com 


65 


4 


8 


15 


Chronicle of Higher 
Education 


193 


4 


7 


9 


Total 


327 


9 


29 


44 



This finding bears out our informal observations of how people tend to talk about jobs, particularly in adult-learning 
settings. 

Titles of Academic Programs 

We also felt it would be useful to see what universities are labeling the degrees and programs because this 
data could perhaps be the most persuasive in influencing graduates perceptions of the field. We utilized the 
Curricula Data Of Degree Programs In Educational Communications And Technology listed on the AECT’s 
website to identify Universities who had programs of study in our field. Even though this lists does not include 
every possible program throughout the country - mainly because universities have to self-select to be included on 
this list by submitting the required information - it does list the majority of programs throughout the country. 
Eurther, there is reason to believe that the programs that are listed represent groups that identify with AECT. 

The Curricula Data Of Degree Programs lists both the title of the degree as well as the title of the program 
at the institution. The list contained 134 programs in the United States. We specifically chose to only sample 
programs in the U.S. because AECT, despite its international influence, is historically and primarily an American 
professional organization. We counted any degree or program that had the words Educational Technology, 
Instructional Technology or Instructional Design in the title. Eor instance, a degree or program called Instructional 
Systems Design would be counted in this category but a program called Instructional Systems would not. Of the 134 
programs, 20 of the programs used the label Educational Technology in the degree name, 29 used the label 
Instructional Technology, and 9 used the label Instructional Design or Instructional Design and Technology. The 
numbers increase when you look at the names of programs (see Table 2 and Eigure 1). There were 34 programs with 
the name Educational Technology, 51 with the name Instructional Technology, and 13 with Instruction Design or 
Instructional Design and Technology in the name. This is most likely due to the fact that as hard as it is to change 
the name of a program or department, it is even harder to change the name of a degree. 

Table 2 

Breakdown of the Labels Used by Universities 





Degree Name 


Program Name 


Educational Technology 


20 


34 


Instructional Technology 


29 


51 


Instructional Design and/or 


9 


13 


Instructional Design & Technology 
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Figure 1. Visual Depiction of Labels Used by Universities 



By Degree Name 




Educational Technology 



■ Instructional Technology 



- Instructional Design or 
Instructional Design and 
Technology 

- Other 



By Program Name 




Educational Technology 



■ Instructional Technology 



- Instructional Design or 
Instructional Design and 
Technology 

^ Other 



Listserv Conversation 

Issues of naming conventions and labels are regularly brought up on listservs like ITFORUM and DEOS. 
Listserv participants occasionally argue for new labels like learning design or learning technology. Advocates of 
educational technology, however, seldom surface. We did a few basic searches in the ITFORUM archives, using 
key words like “Definition of the field” and “Labels used in the field,” to get a sense of some of the past discussions 
that have taken place. As we suspected, we did not find anyone advocating for the use of the label educational 
technology. However, a few themes did emerge. 



301 



First, members of the ITFORUM acknowledge that our current labels are confusing. The following excerpt 
is representative: 

... terms 'education technology' and 'instructional technology' are also a little ambiguous to newcomers 
because of the uncertain relationship between the two words - is it the technology *of* instruction, or is it 
instruction *about* technology? Many assume it is the latter because they know that 'children learn about 
computers in schools'. 

But despite dissatisfaction with the labels educational technology and instructional technology, members of 
ITFORUM cannot agree on an alternative label. Some are not happy with the label instructional designer: 

We are ... more than "instructional" designers because that limits us and gives the perception of designing 
for the "giving" of instruction. In the times and days of constructivism e-learning, "blended" learning etc. 
the main word is learning. I agree that it should be along the Learning Systems Design to more adequately 
describe what we do ... That will be the biggest hurdle, since even changing a program name in academia 
takes forever . . . 

. . .My problem with instructional designer is they want me to create stand-up training. When I use 
instructional technologist, I get blank looks (from, I think, confusion with Information Technology). When I 
talk about educational technology, I get asked to wire classrooms. ... I agree that Instructional 
Technology has the unfortunate overlap with the other IT acronym. However, I also think Instructional 
Designer doesn’t separate us as the ones who play with toys. ... I’m trying to make us think beyond just 
instruction, to other forms of learning, to start taking responsibility for performance support, knowledge 
management, and other information needs that lead to ability to act (see my forthcoming Educational 
Technology article, he says in blatant self-promotion), so I’ve been looking for a new term. Given that 
instruction is only a part of that, I think at least we should think about learning technology, or be willing to 
be even broader (learning technology and performance support...), or something. I have no simple answer, 
I confess, so I’m continuing to troll for a new branding. 

. . . asking for a definition of IT is not the right way to go about understanding the eclectic nature of our 
field. I think that the corporate world has come the closest to pinning down the term "instructional 
designer" as an accepted position description .... lam not sure though that as an academic IT community 
we ought to seek a solid definition for the field. If we were able to do that, it would cease to be a definition. 

Others have mixed feelings about the role of “performance”: 

...I think that whatever the field is called it will be for our own organization and purposes. Whether I tell 
someone I'm an ID&T, IT or HPT - the general public will still be saying, "Now what is it you. ...I have 
nothing against the inclusion of non-instructional solution to the tasks ... / just found it a little bit ironical 
to ... use the name Instructional Design and Technology for our field when ...we already expanded our 
scope covering even the non-instructional solution like improving incentive or rewards system for 
employees, why not totally change the name of our field to something that woudfit to its actual nature. So, 
if Human Performance Technology would best describe our field , so be it . Because sometimes I'm just 
confuse to refer our field , ID&T or HPT? 

...I personally do not like the word performance in the name because of a personal adversion to indicating 
that I deal with making others "perform. " It gives me a feeling of a monkey on a chain and it is but one 
aspect of our skills. ...I prefer the word "development. " 

Others seem to struggle with AECT’s place in our field, specially given other professional organizations. Take for 
instance the following posts: 

... AECT released a new definition of educational technology last ... they specifically used educational 
technology because that was the term used in the name of the organization. Based on that definition, who 
should be included as programs in our field? Or should we be using AECT as our measuring stick? ...If 
not AECT, what is our primary professional organization? If we have more than one primary professional 
organization, are we really all in the same field? 
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...Part of the problem with the definitions is that AECT has been less than consistent, too media related 
and there is no professional group of IDs that defines the profession or IT and we have argued here over a 
name(see the archives and papers). AECT still caters too much to kl2 and NCATE accredited programs 
ignore the business, military, govt and training world where many IDs work. 

...There are clearly more than three of four organizations that inform the field. Off the top of my head, I 
can think of eight: AECT, AACE, ISPI, ITSE, ASTD, IEEE, ACM, and Educause. While each of these 
organizations takes a unique perspective (or at least tries to), they all contribute something to the research 
and practice of our field. I think it would be disingenuous to not recognize that.... 

Some have pointed out that given the diversity of our field, any label will leave some out: 

. . .given the broad scope of our filed, regardless what label we use. . ., we will leave out some “important” 
aspects of what we do (or at least what we think we do). At least for now, I am not sure how productive it is 
to try to derive one “right” label for our field. I do agree with Clark, ideally, that “we do need a good label 
for this group, for branding purposes ”. However, I am not optimistic that we ( the filed) are ready to agree 
on one label. . . 

Publications 

The last indicator we looked at to get a sense of how people in our field our labeling or referring to our 
field was publications— specifically, books. This was perhaps the least systematic of our analyses but we wanted to 
point out a few trends that we have noticed regarding publications in our field. 

Daniel Surry and colleagues took a list of 700 books in our field and had surveyed people on their 
perceptions of the most influential / foundational. Of the list that they identified, only one book. Trends & Issues in 
Educational Technology (Ely, 1989) had the label educational technology in the title. Surprisingly though, only two 
books had instructional technology in the title. They were Instructional Technology: The Definition and Domains of 
the Field (Seels & Richey, 1994) and Classic Writings on Instructional Technology (Ely & Plomp, 2001). However, 
there were over 19 books with instructional design in the title. 

Concluding Thoughts 

Our critique of the new label leaves certain issues unexplored (e.g., the political subtexts and competitive 
environment that the definitions committee worked within). Regrettably, these issues remain unexamined and un- 
argued for. There may be issues and requirements facing AECT and the field that we are unaware of. We therefore 
invite a response from the terminology committee to our paper, and more open discussion of these issues that will 
shape our professional practice for years to come. 

AECT is the oldest of the educational/instructional technology organizations. Throughout its history, it has 
continually changed with the times. Eor instance, AECT began as the Department of Visual Instruction-, then it 
changed its name lo Audiovisual Instruction and then to Audiovisual Communications (Torkenson, 1998), and then 
later in the early 1970s to Association for Educational Communications and Technology (AECT) (Molenda, 2005). 
Another example of AECT changing with the times is when in the late 1980s it merged two journals to create 
ETRD. So throughout its history, AECT has not been afraid to make changes and change with the times. 

Despite its changes and its history, AECT appears to be losing market share to competing professional 
organizations — both in terms of overall numbers as well as segments of its population. For instance, regarding 
overall numbers, in 1998, there were 5,280 individual members in AECT (Pershing & Lee, 1999); but in 2006, it 
had only 2,200 individual members (Pershing, Ryan, Harlin, & Hammond, 2006, p. 1 1). Further, in terms of a shift 
in population, in the late 1980s and 1990s, the percentage of members working in K12 started to decrease (Pershing 
et ah, 2006). During this same time, the number of members focused on business and industry began to increase 
(Torkelson, 1998), but not enough to make up for the loss of K12 focused members. Now in 2006, the majority 
(perhaps 70%) of AECT's members work in higher education (Pershing et ah, 2006). 

These trends in membership are even more troubling when one considers the fact that ISTE has an 
estimated 18,000 members (ISTE Annual Report, 2008). ISTE is not AECT’s only competitor. ASTD, ISPI, AACE, 
SLOAN-C, and Educause all compete for a similar member base (See Figure 2). Thus, even though AECT is the 
oldest of the organizations, and once had a membership of over 9,000 in 1970, it is now one of —if not the smallest- 
professional organization in our field. 
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Figure 2. Membership Comparison 




As an organization, AECT needs to focus on its future. This is not something new though; throughout its history, 
AECT has had to adjust to the times. Almost 20 years ago, Reigeluth (1989) claimed that “the field is undergoing an 
identity crisis like none in its history” (p. 67). With the rise of online learning and the increased presence of 
competing, and at times more specialized professional organizations, AECT finds itself in many ways once again in 
an identity crisis. We suspect that part of AECT’s problem might be the “vague and inconsistent language” that 
Reigeluth and Carr-Chellman (2006) explain can impede a disciplines growth. Morgan even pointed out in 1978 that 
“some would say that a discipline about whose name there is no certainty is no discipline at all, and educational 
technology has a variety of other labels — instructional systems development, instructional design, and, occasionally, 
educational engineering” (p. 142). 

So rather than revert to dated labels that are being used less and less and continue encouraging using terms 
interchangeably, we posit that AECT should have simply considered using the label instructional technology or if 
any change was to be made, it should have put instructional design front and center and consider using a new label 
that highlights the role of design. While there is a growing number of people who recommend adopting a label such 
as “Learning Design,” we recommend adopting what Reiser and colleagues (e.g.. Reiser, 2007) have been 
suggesting for years, the use of instructional design and technology . Instructional Design and Technology places 
instructional design practice in the forefront while maintaining the technology focus. We recognize that many see 
instructional design as simply a part of the field of instructional technology (e.g., Moldena, 1997) or even as its own 
field (Wilson, 2005); we believe however that there is a design component in nearly every aspect of instructional 
technology. Therefore, while Heinich (1984) put technology as the base of our field in 1984, we believe that design 
is equally important and should have equal standing. Reiser (1987) has pointed out that there have been basically 
two types of definitions of our field over the years, the type that focuses on technology and media and then the type 
that focuses on systematic design. We believe, especially with the increased pressure of other professional 
organizations, that AECT needs to begin to differentiate itself from its peers. Reiser’s label also has the advantage of 
a more unique acronym (IDT) than other proposed labels, useful as a shorthand reference. 

We recognize that any label, even Instructional Design and Technology, will unquestionably highlight 
something’s while ignore others. Or as Saun Shewanown pointed out on March, 25, 2004 on ITEORUM; “regardless 
what label we use for ourselves, we will leave out some important aspects of what we do (or at least what we think 
we do).” We even agree to some degree with Saun, who stated in the same message, “I am not sure how productive 
it is to try to derive one right label for our field.” We don’t think we can ever find the “right” or the “perfect” label 
for the field. But we might be able to find a better label than reverting to an outdated one such as educational 
technology. Eurther, the question might not be so much one of defining the field but defining AECT’s position in the 
field. 
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We conclude with the words of Ely and Plomp (2001), who have encouraged dialogue such as that found in 
this paper; 

Where are the voices today? What are they saying? How has the field changed? How does it need to 
change? Professionals should continue to seek answers to such questions if the profession is to grow 
and prosper? (p. 255) 
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Introduction 

In July 2005, industry leaders and the U.S. Chamber of Commerce expressed “deep concern about the United States' 
ability to sustain its scientific and technological superiority through this decade and beyond” (United States 
Department of Commerce, 2005). One of the issues is that American students are ill prepared to enter careers in 
science, technology, mathematics, and engineering (STEM). The dissatisfaction with American education is shared 
by the public. In a Gallup poll conducted in August 2005, over 50% of the respondents were either “completely 
dissatisfied” or “somewhat dissatisfied” with American schooling. The Federation of American Scientists (FAS) 
released a report in October 2006, proposing digital games as a solution to reshape education. The report was based 
on the work of a National Summit on Educational Games convened in October 2005, which brought together 100 
experts in various fields to discuss approaches to accelerate the development, commercialization, and deployment of 
new generation games for learning. At the same time when the FAS report was released, the MacArthur Foundation 
(2006) launched a $50 million dollar initiative with a focus on digital media and learning, including electronic 
games. 

In spite of the growing interest in developing electronic educational games, there is little guidance on how to design 
effective electronic educational games. Educators have little experience in designing electronic games and the 
computer game industry lacks expertise in integrating effective pedagogy into game design. The type of educational 
computer games that schools and the game industry are familiar with is edutainment, which usually focuses on the 
motivational effects of games and typically employs game-like drill and practice activities to achieve lower level 
learning goals. Many of the edutainment endeavors failed to produce anything either educational or entertaining 
(Kirriemuir & McFarlane, 2003; Okan, 2003). Theory and research is needed to inform designers of electronic 
educational games. This paper shares our findings in a formative evaluation of an electronic game currently under 
development. The findings may inform future development of electronic game and add to the body of knowledge on 
learning in electronic game-based learning environments. 

Project Background: Conquest of the Coastlands 



Back story 

Conquest of the Coastlands (CoC) is a role-playing game with a science fiction/fantasy setting. It was a 3-D game 
developed using Torque game engine. Pursuing larger strategic objectives, the player character will be challenged 
with a variety of problem solving “quests,” which form the main plotlines of the interactive narrative and will 
provide the immediate motivations for player-character activities in the game. Each quest is designed to achieve 
specific learning goals. 

The game takes place amid an ancient conflict between two sentient species and their struggle for dominance on a 
planet in another solar system. While not technologically sophisticated, the planet’s two rival sentient species have 
reached a turning point in their evolutionary history where it is likely that one — the Mruk-ma — will likely drive the 
other — the Sheft-ma — into extinction. 
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The Mruk-ma are an aggressive, sea-faring species, while the Sheft-ma are city-builders who make their home in 
“The Coastlands,” along the marshy seashores and river valleys of Mertis’ lone continent. For the vulnerable Sheft- 
ma, the strategic key to their self-defense is a deteriorating system of fortifications built in the coastal wetlands 
surrounding their cities. But these wetlands are mysteriously disappearing at an alarming rate, and the threat of 
invasion by Mruk-ma fleets is growing. 

A decisive change comes when the survey ship of an advanced alien race crash-lands in the oceans of Mertis. 
Arriving in escape pods from their doomed spaceship, the strangers, called Cilati, are scattered around the planet. 
Now hopelessly stranded on Mertis, some of the alien crew manages to make their way to The Coastlands, where 
they are warmly welcomed by the Sheft-ma. The Cilati survey team brings with them precious scientific knowledge, 
technology, and methods that could dramatically shift the balance of power in the conflict between the two rival 
species. The survival of the Sheft-ma will depend on whether they can effectively utilize the science and tools of the 
Cilati to rebuild their crumbling forts and defend their disappearing coastlines. 

The Cilatis are a highly advanced race of space-faring explorers. Extremely long-lived, they traverse the galaxy in 
pursuit of knowledge about other planets and other life forms. Cilati ships have visited countless worlds, quietly 
observing the species that inhabit them. Generally, they never interfere in the cultures they study, and they seldom 
even make their presence known. 

However, it quickly becomes apparent that the Mruk-ma have adopted a radically new strategy in their struggle with 
the Sheft-ma: ecological warfare. By attacking the delicate environment on which their peaceful rivals depend, the 
Mruk-ma hope to wreck the Sheft-ma civilization and eliminate their species. 

Theoretical Framework and Quest Design 

Figure 1 illustrates the program theory underlying the lesson that we created. We expect improved learning outcome 
in students’ life science knowledge. To achieve this goal, we designed the game based on related theory and 
research. For example, to facilitate the acquisition of life science knowledge, we adopted instructional strategies 
suggested by the four component instructional design model (van Merrienboer, Clark, & de Croock, 2002) and 
cognitive apprenticeship (Collins, Brown, & Holum, 1991). Research shows that intrinsic motivation (Malone & 
Lepper, 1987) and flow (Csikszentmihalyi, 1991) positively contribute to learning (Cordova & Lepper, 1996; 
Csikszentmihalyi, 1991; Hektner & Csikszentmihalyi, 1996). We adopted various motivational strategies such as 
developing meaningful challenges, providing choices and personalization opportunities, enhancing sensory and 
cognitive curiosity (Malone & Lepper, 1987). Our engagement design was also informed by theory and research 
related to flow, which is the sense of control, deep engagement and exhilaration when one is involved in an optimal 
experience (Csikszentmihalyi, 1991). Previous research on flow suggests that flow antecedents such as challenge- 
skill balance, clear goals, unambiguous feedback, playability, gamefulness, and a frame story may contribute to the 
experience of flow, which provide guidance on design features for educational games (Kiili & Lainema, 2008). We 
incorporated these features into our game. 




Figure 1. Program Theory of Conquest of Coastlands. 
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The lesson we evaluated in this study includes a quest and several debriefing sessions. In CoC, a quest is a journey 
which the player character takes on to complete a goal or a task, which typically involves the completion of a 
scientific investigation. On each journey, the player character navigates the game space and works in teams to 
achieve the quest goal with the assistance of NPCs, tools, and scaffolds. In this study, we evaluated the glim quest. It 
is a four-hour life science and environmental science lesson for children aged 1 1-13. The lesson consists of 
approximately two and a half hours of electronic-game activities followed by approximately half an hour of 
debriefing. Pretest and posttest took about one hour. The primary goal of this lesson is to teach an integrated set of 
knowledge and skills that allow students to address environmental issues. The quest begins with the player character 
being summoned before the Council of Elders. The player character is charged with researching how the 
introduction of a strange creature, which was introduced to the coastal marshlands by the Mruk-ma species as part of 
the ecological war, is related to the depletion of glim, a key staple of the Sheft-ma diet. As the quest unfolds, the 
player character learns more about the attributes of the strange creature. They identify a set of complex factors that 
connect the creature and the fish, including networks of predator/prey relationships and the relationship between the 
environment and the species. This helps students understand that introducing a competitor not only impacts species 
that are closely linked to each other on the food web, but also affects those that are not directly related. This module 
addresses the concepts and principles related to population and ecosystem, a key learning outcome in Content 
Standard C, grades 5-8 (National Research Council, 1996). The following is the structure of the quest that we 
created. 

• Opening cinematic: A pre -rendered cinematic was created to set the stage for the game. The opening cinematic 
introduces the characters and setting, and provides hints about the overall goal of the game: to preserve and 
restore the fragile ecosystems in the coastlands. 

• Character creation: Character development is a key component of role playing games (RPGs). Character 
creation is the first step toward developing characters. In the current version of the game, the learner may create 
their player character by choosing gender and giving it a name. 

• Completion of the quest. The quest consists of: 

0 Quest goal/task: The quest goal is presented by a short cinematic, depicting how the player character is 
commissioned with the task to investigate the causes of fish depletion. 

0 Quest encounters: In RPGs, quest encounters usually involve interaction with a non-player character 
(NPC). In the glim quest, encounters are designed to facilitate one or more of the following functions: 
delivery of information or a directive, engagement of the player character in a particular activity, and 
procurement of an item. 

0 Tools and resources: Tools commonly found in commercial games, such as health bars and localized maps 
are available in CoC to provide access to different locations in the game space and to offer feedback on the 
status of the game. A personal digital assistant (PDA) serves as the primary tool for the player character to 
complete the quest. It facilitates communication between the learner and the mentor, allows data collection 
and analysis, and enables presentation of case studies and other related resources. 

0 Case study scaffolds: A set of case studies are provided to help learners solve the mystery of fish depletion. 
The case studies may facilitate understanding of predator/prey relationships and the relationships between 
the environment and the species. These case studies are provided during the quest encounters. For example, 
a case study of dead zone is presented to help the PC learn the relationship between farm runoffs, dissolved 
oxygen, fish depletion, and algae bloom. In a Socratic dialog, the alien mentor asks the PC a series of 
questions to attend to important details in the case and to analyze the relationship between various factors 
described in the case study. 

0 Modeling: Modeling is embedded in the game to support the scientific inquiry process and to help learners 
use scientific language to create research plans and communicate research findings. In our previous work 
on the game, we found that students did not acquire the skills from the scientific inquiry tutorial embedded 
in the game to successfully complete the quest. Therefore, we developed modeling and coaching scaffolds 
to guide the scientific inquiry process. For example, at the beginning of the inquiry process, PCs are asked 
to create a data collection plan; during the process, they write a progress report from time to time; toward 
the end of the process, they develop a final report in order to draw conclusions and discuss findings. 

Research Design 

Design-based research serves as a framework to guide our research effort. Design-based research emphasizes the 
design of innovative learning environments based on theory and empirical research through an iterative process of 
design, implementation, analysis, and redesign (Design-Based Research Collective, 2003). The purpose of this 
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iterative process is not only to enhance the particular intervention being investigated but also to develop theories to 
account for the impact of the intervention and to create models to inform the design of other innovations. Design- 
based research requires collaboration among practitioners, researchers, and technologists (Reeves, Herrington, & 
Oliver, 2004). 

Qualitative and quantitative methods may be appropriate for different stages of design-based research (Bannan- 
Ritland, 2003; Design-Based Research Collective, 2003; Kelly, 2004; Shavelson, Phillips, Towne, & Feuer, 2003). 
Qualitative methods are especially helpful during the exploration phase of a design-based research project (Kelly, 
2004; Shavelson et ah, 2003) when models (Sloane & Gorard, 2003), conjectures (Sandoval, 2004), or hypothesis 
(Kelly, 2004) are formulated in the context of real world problems and interventions. Quantitative methods are 
helpful later when knowledge that emerged from qualitative explorations is validated. Both quantitative and 
qualitative methods were employed in this study to assess and explain the impact of the game. 

Research Questions 

The purpose of this research is to conduct formative evaluation of the glim quest to determine the effectiveness of 
the game design so as to generate design principles that may inform the development of this and similar games. 

• Question 1 : Do student participants demonstrate improved performance on life science knowledge measure? 

• Question 2: How well do student participants experience flow in the game? 

• Question 3: How do students perceive the game? 

Participants 

Twenty children from grades five to eight participated in the study at a Southern research/teaching university. 
Among the students, fourteen were boys and six were girls. The participants have diverse ethnic backgrounds. Then 
students identified themselves as White Caucasians; eight students identified themselves as of Asian origin; two 
students identified themselves as African Americans. Seventeen students are in gifted programs. Two students are in 
a private school. One student is in a regular program in a public school. These students were recruited by word of 
mouth. 

Data sources 

A knowledge measure was a 15-item test developed by our research team to answer research question one. This 
measure of knowledge and skills employs a couple of assessment methods, as articulated by Wiggins and McTighe 
(1998), to include objective measures and academic prompts. Objective measures assess recall of factual 
information and concepts through selected response and short-answer prompts. Academic prompts assess critical 
thinking and problem solving through open ended problems with multiple solutions. 

A GameFlow questionnaire (Kiili & Lainema, 2008) was adopted to examine how well students experience flow in 
the game. The questionnaire can be found online at http://amc2.pori. tut.fi:8080/Forms/realgame/survevl.html . The 
questionnaire consists of 36, five-point, Likert scale questions that measure three flow dimensions proposed by 
Csikszentmihalyi (1991): Flow antecedents, indicators of flow state, and flow consequences. Flow antecedents are 
qualifying factors or prerequisites that contribute to the experience of flow. The questionnaire measures the 
following antecedents: challenge-skill balance, clear goals, unambiguous feedback, playability, gamefulness, and a 
frame story. Indicators of flow state include concentration, autotelic experience, time distortion, sense of control, 
and loss of self-consciousness. Flow consequences include exploratory behavior. Previous research on the 
GameFlow questionnaire (Kiili & Lainema, 2008) shows that the reliability estimates for most of the indicators of 
flow are relatively high (from .75 to .85). The reliability estimates for some of the flow antecedents are higher, 
including unambiguous feedback (.68), sense of control (.79), playability (.78), and a frame story (.88). The 
reliability estimates for some other flow antecedents are lower, including challenge-skill balance (.56), clear goal 
(.57), and gamefulness (.37). The reliability estimate for the flow consequence factor, exploratory behavior, was low 
too (.22). 

Focus groups of students were assembled to provide in-depth perceptions of the game. Students talked about what 
they liked or disliked about different aspects of the game, including the characters in the game, advices and guidance 
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provided by the characters, case studies, and feedback. They also discussed how easy or difficult it was to play the 
game and provided suggestions for future changes. 

Procedure 

Prior to the lesson, the knowledge measure was given to students as the pretest. They spent 20 minutes or less to 
complete the pretest. The lesson that followed was broken up into four sessions. Each session included activities in 
the game environment followed by a brief debriefing session. Debriefing helped students draw out the key 
information delivered in the cinematic, understand the important concepts taught in the game, and provide feedback 
to students. Debriefing was included as a part of the lesson, because previous research shows that debriefing plays a 
critical role in helping students make the connection between game playing and scientific concepts (Egenfeldt- 
Nielsen, 2005). After the lesson, the knowledge measure was given to students again as a posttest and the 
GameElow questionnaire was also administered. Students spent about 30 minutes or less to complete the posttest 
and the questionnaire. 

Data analysis 

In order to answer research question one, “Do student participants demonstrate improved performance on life 
science content knowledge and inquiry skills?”, a one-tailed, paired t-test was calculated comparing the pretest and 
posttest scores on life science content knowledge and inquiry skills. To answer question 2, “How well do student 
participants experience flow in the game?”, descriptive statistics were calculated for multiple flow dimensions, 
including 1) flow antecedents such as challenge-skill balance, clear goals, unambiguous feedback, playability, 
gamefulness, and frame story, 2) indicator of flow state, such as concentration, autotelic experience, time distortion, 
sense of control, and loss of self-consciousness, and 3) flow consequence including exploratory behavior. To answer 
question three, “How do students perceive the game?” The focus group interviews were recorded and transcribed. 
Miles and Huberman's (1994) data analysis procedures were followed to analyze the qualitative data. We first coded 
the transcripts and journals into conceptual chunks and grouped the chunks into categories. Then, we ran queries to 
make sense of the relationship among the categories. Lastly, we wrote conclusions to explain the quantitative results. 

Eindings 

Question 1 : Do student participants demonstrate improved performance on life science knowledge measure? 

Statistical analysis indicates a significant difference on the life science knowledge measure from pretest to posttest 
t(20) = -8.51, p = .000 (M pre = 34.26; M post = 56.68). That is, the game had a statistically significant impact on 
students’ life science knowledge. 

Question 2; How well do student participants experience flow in the game? 

In Table 1, the high mean values of the indicators of flow and consequence of flow show that students experienced 
flow while playing the game. The mean values of the flow antecedents indicate that the game was well designed and 
provided appropriate circumstances for players to experience flow. 

Table 1. Mean scores, standard deviations of flow dimensions (N = 20). 



Elow Dimension 


Mean 


Standard Deviation 




Elow antecedent 


Challenge-skill balance 


3.72 


0.91 


Clear goals 


4.3 


0.48 


Unambiguous feedback 


4.22 


0.68 
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Playability 


3.57 


0.80 


Gamefulness 


3.47 


0.89 


Frame story 


4.17 


0.66 




Indicator of Flow 




Concentration 


3.78 


1.12 


Autotelic experience 


3.87 


0.93 


Time distortion 


3.75 


0.94 


Sense of control 


4.03 


0.62 


Loss of self- 
consciousness 


3.62 


1.05 




Flow Consequence 




Exploratory behavior 


3.62 


0.89 



Question 3; How do students perceive the game? 

In general, students had positive perceptions of the game. They thought it would be fun to use it in school. They 
seemed to think that the pedagogical aspects of the game were well designed. Students thought that the game 
challenged them to think. The case studies, feedback, and guidance provided by the mentors in the game were 
helpful. They would like more explanation in the feedback. They liked the PDA, which is a versatile tool that 
facilitates data collection and enables communication between the mentor and the student. The only concern they 
had was that the interaction with a mentor about water quality was long and slow. They believed that more 
encounters with other characters and more mini quests might help make the game more interesting. 

The game aspects of the glim quest were less satisfactory to students. First of all, there were technical glitches. For 
example, the game crashed for a few students and they had to restart the game. Because the game did not save the 
location of the encounter at a very granular level, students had to play from the beginning of the encounter where 
their game crashed. They had to repeat some of the activities. Moreover, students compared this game with 
commercial games that they are used to playing and expressed many concerns. We were aware of many of these 
issues prior to the evaluation. We have not had the time or resources to fix these issues yet. For example, students 
commented that the movements of some game characters need improvement. We had difficulties addressing this 
issue because of the Torque game engine. The animations of the game characters look fine in 3D Studio Max, a 3D 
modeling tool. However, the movements of the characters were distorted after they were imported into Torque game 
engine. Students expected the typical commercial game features including levels, easy navigation using maps, and 
health bar. Some of these features were partly implemented in the current game, but their complete implementation 
requires more time. Another issue relates to the implementation of the game characters. One of the alien characters 
in the game does not look appealing to some students. The personality of an uncle character also annoyed some 
students, because the character scolded students for mistakes made because of the lack of prior knowledge of the 
context of the story. Modification of these characters might be needed. 
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Conclusions and Discussions 



The formative evaluation of the glim quest in Conquest of Coastlands shows that students’ life science knowledge 
improved significantly after experiencing the game-based lesson. Control group will be needed in future studies to 
compare the effectiveness of the game-based lesson with lessons which teach the same content but delivered without 
the 3D game environment. Larger sample size is also needed in the future to determine the effectiveness of the 
lesson. 

Results from the GameFlow questionnaire indicate that students seem to have experienced some level of flow in the 
game-based lesson. Previous studies (Csikszentmihalyi, 1991; Kiili & Lainema, 2008) suggest that flow may 
contribute to learning. Although our evaluation is not rigorous enough for us to claim that students’ flow experience 
in this game enhanced their learning, the improvement of students’ life science knowledge and high mean values on 
the GameFlow questionnaire suggests that the current study is at least consistent with previous finding concerning 
the correlation of flow and learning. Again, future studies will be needed to compare the game-based lesson with 
traditional lessons in terms of students’ flow experience. Furthermore, more work is needed to improve the 
reliability and validity of the GameFlow questionnaire. 

Focus group interviews help understand students’ experience in the game-based lesson. Overall, students had 
positive perceptions of the game and the pedagogical aspects of the game seem to be well designed. This may 
explain students’ learning gains and findings on the GameFlow questionnaire. This is encouraging, because it 
indicates that game-based lessons have the potential of impacting student learning. The game aspects of the lesson 
were less satisfactory when students compared the environment to commercial games. This was not surprising. 

Game companies spend millions of dollars on commercial game. Students will be disappointed if they expect 
educational games, especially those produced with a small budget, to have the same production quality as 
commercial games. Establishing appropriate expectation is important for students prior to the game playing 
experience. However, this does not mean that educational researchers should live with low quality production 
quality for educational games. The difficulties our team has encountered might be partly attributed to the game 
engine that we have chosen. New and more powerful game engines have emerged in the market since the start of our 
project. Exploring other game engines might help us address the various technical issues reported by the students. 

References 

Bannan-Ritland, B. (2003). The role of design in research; The integrative learning design framework. Educational 
Researcher, 32(1), 21-24. 

Collins, A., Brown, J. S., & Holum, A. (1991). Cognitive apprenticeship: Making thinking visible. American 
Educator, 6-11, 38-46. 

Cordova, D. I., & Lepper, M. R. (1996). Intrinsic motivation and the process of learning; Beneficial effects of 
contextualization, personalization, and choice. Journal of Educational Psychology, 88(4), 715. 

Csikszentmihalyi, M. (1991). Flow: The psychology of optimal experience. NY; Harper Perennial. 

Design-Based Research Collective. (2003). Design-based research; An emerging paradigm for educational inquiry. 
Educational Researcher, 32(1), 5-8. 

Egenfeldt-Nielsen, S. (2005). Beyond edutainment: Exploring the educational potential of computer games. 
University of Copenhagen, Copenhagen. 

Hektner, J. M., & Csikszentmihalyi, M. (1996). A longitudinal exploration of flow and intrinsic motivation in 

adolescents. Paper presented at the annual meeting of the American Educational Research Association, 

New York. 

Kelly, A. (2004). Design research in education; Yes, but is it methodological? Journal of the Learning Sciences, 
13(1), 115-128. 

Kiili, K., & Lainema, T. (2008). Eoundation for measuring engagement in educational games. Journal of Interactive 
Learning Research, 19 (3), 469-488. 



313 




Kirriemuir, J., & McFarlane, A. (2003). Literature review in games and learning. Retrieved October 10, 2005, from 
http://www.nestafuturelab.org/research/reviews/Q8 01 .htm 

MacArthur Foundation. (2006). Digital media and learning [Electronic Version]. Retrieved March 2, 2007 from 

http://www.digitalleaming.macfound. 0 rg/site/c. enJLKONlFiG/b.2029199/k.BFC9/Home.htm?&auid=2276 
796 . 

Malone, T. W., & Lepper, M. R. (1987). Making learning fun: A taxonomy of intrinsic motivations for learning In 
R. E. Snow & M. J. Earr (Eds.), Aptitude, learning and instruction: III. Conative and affective process 
analyses (pp. 223-253). Hillsdale, NJ: Erlbaum. 

Miles, M. B., & Huberman, A. M. (1994). Qualitative data analysis: An expanded source book (2nd ed.). Thousand 
Oaks, CA: Sage Publications. 

National Research Council. (1996). National science education standards. Washington, DC: National Academy 
Press. 

Okan, Z. (2003). Edutainment: Is learning at risk? British Journal of Educational Technology, 54(3), 255-264. 

Reeves, T. C., Herrington, J., & Oliver, R. (2004). A development research agenda for online collaborative learning. 
Educational Technology Research and Development, 52(4), 53-65. 

Sandoval, W. A. (2004). Developing learning theory by refining conjectures embodied in educational designs. 
Educational Psychologist, 59(4), 213-223. 

Shavelson, R. J., Phillips, D. C., Towne, L., & Eeuer, M. J. (2003). On the science of education design studies. 
Educational Researcher, 52(1), 25-28. 

Sloane, E. C., & Gorard, S. (2003). Exploring modeling aspects of design experiments. Educational Researcher, 
52(1), 29-31. 

van Merrienboer, J. J. G., Clark, R. E., & de Croock, M. B. M. (2002). Blueprints for complex learning: The 4C/ID- 
Model. Educational Technology Research and Development, 50(2). 

Wiggins, G., & McTighe, J. (1998). Understanding by design. Alexandria, VA: Association for Supervision and 
Curriculum Development. 



314 



Making a Significant Difference: A Goal-Driven Approach to Improving 
Teaching & Learning with Technology 

Jon Mott 

Brigham Young University 
D-367 ASB 
Provo, UT, 84602 

Whitney McGowan 
The Center for Teaching & Learning 
3800 HBLL 
Provo, UT, 84602 

Larry Seawright 

The Center for Teaching & Learning 
3813 HBLL 
Provo, UT, 84602 

Stephanie Allen 

The Church of Jesus Christ of Latter-day Saints 
3800 HBLL 
Provo, UT, 84602 

Descriptors: Sloan Pillars, design-based research 
Introduction 

As innovative media technologies appear on the educational scene and are adapted for instructional 
purposes, a promise of “new and improved” teaching and learning seems to accompany each advancement. With 
this hope in mind, research studies are conducted to test the application of these new technologies against other 
media technologies and teaching methods. Throughout the years as these studies have been conducted to compare 
the effectiveness of one technology over another, the results have almost always been reported as “No Significant 
Difference.” 

During the 1980s when many of these “No Significant Difference” outcomes emerged, Thomas L. Russell 
(1999) from North Carolina State University conducted an inventory of these comparative studies to identify cases 
in which technology was actually proven to improve learning. He found there were very few studies that attributed a 
significant difference to the application of technology to instruction when compared to traditional methods — that 
“there is nothing inherent in the technologies that elicits improvements in learning.” What he did find was an 
“enormous number of studies” that showed “no significant difference” (p. xii). 

The significance of his “No Significant Difference” discovery was that “no significant difference studies 
provide substantial evidence that technology does not denigrate instruction” (Clark, 1999, p. xiii). He concluded that 
technologies should be used to “increase efficiencies, circumvent obstacles, bridge distances, and the like” (p. xiii). 

Overview of Design-Based Research and Five Pillars 

In summary, the “No Significant Difference” phenomenon highlights that research approaches utilizing 
controlled experiments to “prove” learning gains generally fail to do so. We suggest that a design-based research 
approach may show that technology can, in fact, make a significant difference if the right research assumptions are 
made. The definition of design-based research is: 
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“A systematic but flexible methodology aimed to improve educational practices through iterative analysis, design, 
development, and implementation, based on collaboration among researchers and practitioners in real-world 
settings, and leading to contextually-sensitive design principles and theories” (Design-based Research). 

When employing design-based research, design and theories are developed together in order to improve the 
likelihood that an understanding of what is being tested actually reflects real world results. This method of 
development/experimentation is better for effecting change because it takes into account many factors instead of 
attempting to isolate one or two. Thus the results tend to provide a more comprehensive picture of reality. 

Because design-based research takes more variables into account as the design is iterated, it can be more 
thorough but it also takes a longer time. Theories and interventions are created and tested continuously because they 
are being developed as the research is performed. The nature of this type of research also allows for more flexibility. 
Since the approach incorporates many different methods, it tends to produce more valid data, which in turn, 
increases credibility. But more importantly, the wealth of new information that results from design-based research 
provides a better foundation for enacting effective change in the teaching and learning environment. 

This potential led us to utilize design-based research instead of traditional methods in the design and 
evaluation of instructional design projects at Brigham Young University The sort of studies that yield “no significant 
difference” findings are typically based on outcomes from controlled and traditional methods of assessment, such as 
standardized tests. However, much more goes tends to go into the typical learning experience than such assessments. 
Results from such limited methods can’t possibly measure all that a student has gained from taking a class, either 
online or in the classroom. 

To assist us in our design-based research approach to demonstrate that technology implementations might 
indeed produce significant differences in learning, we chose to employ the Sloan Consortium’s (Sloan-C’s) Five 
Pillars of Quality Online Education. These five pillars were formulated to help better assess the quality and 
effectiveness of online education but are also well suited for the complex nature of design-based research. While 
originally intended for evaluation of online education, the pillars are also applicable to delivery of any type of 
education, including face to face. 

The five Sloan pillars are: learning effectiveness, cost effectiveness and institutional commitment, access, 
faculty satisfaction, and student satisfaction. Later in this paper, we provide specific examples of projects aimed at 
accomplishing each of these goals. Accordingly, a brief definition of each pillar is in order. 

The first pillar, learning effectiveness, focuses on making sure students are getting a quality education. A 
design-based research approach to learning effectiveness would not look at comparing a face to face course with an 
online course to ensure both student groups achieved similar learning outcomes. Rather, the design-based research 
approach would emphasize iteratively improving a course in a variety of ways over time to improve learning. 

The second pillar, cost effectiveness and institutional commitment focuses on how much is being spent to 
deliver instruction, both monetarily and in faculty time commitments. The ideal is to maximize the ratio between 
learning and resources. 

The goal in the third pillar, access, is to provide students and faculty with the means to use the educational 
tools or experience most effectively. This includes marketing, course access, and learner-appropriate learning 
resources. “Access includes three areas of support: academic (such as tutoring, advising, and library); administrative 
(such as financial aid, and disability support); and technical (such as hardware reliability and uptime, and help 
desk)” (Sloan-C). 

Faculty satisfaction, the fourth pillar, emphasizes the importance of faculty support and job satisfaction for 
instructors. As they seek to help students in a variety of ways, faculty motivation, reward structures and satisfaction 
are important ingredients to a successful educational program. 
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The fifth pillar, student satisfaction, reflects a combination of the first four pillars as they interact to form 
the students’ educational experiences. “The goal is that all students who complete a course express satisfaction with 
course rigor and fairness, with professor and peer interaction, and with support services” (Sloan-C). This last pillar 
is in many ways the most important in examining the education process, especially one that is iteratively designed 
and implemented. 

Sloan-C’ s five pillars provide a helpful framework for rethinking the traditional comparative methods 
approach that so often found “No Significant Difference” between various approaches to the same learning context. 
Using the five pillars, we examine case study in a design-based research approach. In many of the examples, 
focusing on one of the five pillars illuminated flaws in project design, theory and/or method. Accordingly, 
adjustments were made during the design & research processes. By examining case studies for each of the five 
pillars, we provide a viable alternative for demonstrating the significant differences that can be made using 
alternative instructional methods. 



Goals, Strategies, and Tactics 

Far too often, teachers, faculty support staff, instructional designers and media producers begin 
instructional design projects with poorly articulated and vaguely defined goals. This usually results from failing to 
ask a very simple question: What should be better after we complete and implement this project? And just as 
importantly. How we will be able to tell that it is better than it was before? 

By beginning with the end in mind, we can avoid this pitfall and design, produce, implement and evaluate 
instructional design projects within a shared framework of “success.” If all of the stakeholders (particularly the 
instructional designer and the instructor) work together to establish a clear definition of exactly what each 
instructional design project is aimed at improving, every team member is much more likely to stay focused on 
accomplishing that specific goal instead of getting sidetracked with technological intricacies or niceties. 



Below is a simple visualization of this process (Figure 1). While some might view this as a gross 
oversimplification of the traditional ADDIE model, we contend that simplicity yields much needed clarity and focus 
that is often missing in the instructional design process. As the first circle in the diagram suggests, the process 
should always begin with a very clear, unambiguous discussion and definition of currently unmet learning goals. 

Are students under-performing? Are students underprepared to learn? Are teachers floundering in the classroom? 
Does it take too long for the students to learn what they need to learn? Do students and instructors uniformly dislike 
the learning experience? Are some students limited from participating in the learning experience for one reason or 
another? 
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Figure 1: Goal Driven Instructional Design 



Once a clear learning goal is established, instructors, instructional designers and others can devise 
strategies and tactics aimed at accomplishing that goal. For example, if an instructor is concerned that students are 
not mastering foundational principles effectively enough to succeed in the latter part of a course, the goal would be 
to simply help them master those principles before moving on. A strategy of authentic practice and application 
might be deemed appropriate to accomplish this goal. Then specific tactics (e.g. applied assignments, projects, etc.) 
could be designed and implemented to realize the strategy. After implementation, the evaluation would consist of 
simply evaluating the students’ level of mastery of foundational principles in the “old” course versus the “new” one. 
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If preparation is better, the strategy and tactics were effective. Iteratively fine-tuning the newly designed course 
might continuously improve student mastery and performance in the course. 

Examples from the Sloan Pillars 

This section features ten examples of projects using the Sloan Pillars framework (two projects for each 
category). For each example, the initial problem, the solution (how the practice improves one of the five areas), and 
evidence of success are provided. All of the projects were completed at Brigham Young University’s (BYU) Center 
for Teaching & Learning (CTL) (formerly known as the Center for Instructional Design). The CTL is a department 
at BYU that partners with faculty on a variety of levels to help improve teaching and learning. The CTL currently 
supports a broad range of large and small-scale faculty projects to improve on-campus instruction. There are more 
than 35 full-time employees and approximately 115 student employees at the CTL. 

Learning Effectiveness Examples 

Example 1 : BYU Marching Band Drumline 

Problem: The BYU Marching Band Drumline consists of approximately 30 student-musicians from around 
the nation. Each year, the turnover rate in drumline performers at BYU is high due to graduation, EDS mission 
commitments, marriage, and other factors. The musicians also come to BYU with various music technique 
backgrounds, which makes it difficult to develop successful performance of unified percussion techniques among 
the student percussion ensemble. In the past, these music students received a printed copy of the BYU Marching 
Percussion Handbook prior to attending BYU band camp. Although this handbook provided some guidance to the 
students, it only provided words and pictures, which was quite limiting in explaining concepts for a performance 
ensemble. 

Solution: The CTL designed an instructional DVD featuring BYU drumline members demonstrating 
percussion techniques. This DVD proved to be especially effective in helping the students gain uniformity as a 
drumline. Students who use the DVD also acquire a greater comprehension and understanding of the required 
concepts because they can imitate the techniques they watch. 

Results: The DVD is distributed to percussion students many weeks prior to their arrival at marching band 
camp. It has also been utilized by incoming and returning drumline members, as well as education majors. When 
student musicians arrive at BYU to practice before school, they typically have a week or less to prepare for their first 
performance. In the past, the director would have to spend several days teaching the techniques that are found within 
the DVD to obtain uniformity among the musicians. Now when the students arrive at camp, they have mastered the 
basic percussion techniques and are ready to prepare for the pre-game show and the first halftime show. 

Example 2: Virtual ChemLab 

Problem: Chemistry students need access to meaningful laboratory experiences to enable them to connect 
theory with practice, work with solutions and materials, and learn and develop cognitive and analytical skills. 
Unfortunately, this valuable lab time is often limited due to classrooms that have high enrollments and limited lab 
time. 



Solution: To address this critical need for lab access, BYU received funding from the Fund for the 
Improvement of Post-Secondary Education (FIPSE) and created an interactive computer simulation called Virtual 
ChemLab, which helps students learn chemistry through simulations. Virtual ChemLab was designed in 2001 to 
enable students in freshman and sophomore level chemistry classes to have (a) laboratory learning experiences, (b) 
opportunities to reinforce concepts taught in the classroom, (c) to learn creatively, and to (d) demonstrate the logic 
behind experiments that are conducted in the laboratory. The Virtual ChemLab’ s interface is intuitive and easy to 
learn. When students use the simulation they learn to make decisions that would be similar to decisions made in an 
actual laboratory. The outcomes seen in the lab are realistic to the students and are visualized through pictures, 
animations or videos. Professors are also able to easily prepare assignments within the simulation environment and 
evaluate the students based on data captured by the software 
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Results: Thousands of students have used the Virtual ChemLab. Evaluations of the product show that 
students like the simulations and find them to be effective. Students’ knowledge of the subject matter increases, 
which enables them to think more like chemists, and allows them to try different experiments without having to 
worry about wasting chemicals and supplies. They also appreciate that they can reproduce experimental procedures. 
In a study of 1400 Virtual ChemLab users, which included surveys, interviews and observations, researchers found 
that approximately 75 percent of the students said they liked the simulations because they (a) provided them with the 
ability to explore, (b) helped them concentrate on specific chemistry principles, (c) gave them the ability to replicate 
procedures, (d) were easy to use, (e) and provided open-ended learning opportunities (Woodfield, Catlin, Waddoups, 
Moore, Swan, Allen, Bodily, 2004). During the first term the Virtual ChemLab was initiated, students completed 
six related assignments in the class in less than half the time it would have taken to complete these tasks in a 
laboratory. Furthermore, in comparing test results from two semesters (with and without the Virtual ChemLab), the 
final exam scores improved from 77.6 percent to 80.8 percent. 

Cost Effectiveness Examples 



Example 1 : Accounting 200 

Problem: Over the last six years, more than 13,000 students have taken Accounting 200 at BYU. 
Approximately 1200 students take the course each semester. Teaching this course traditionally required the time and 
effort of many faculty members, which took them away from teaching upper division courses where small-class 
sizes are ideal. The primary accounting professor for Accounting 200, Norm Nemrow, wanted to create a more 
effective and efficient way to teach the course, and at the same time, save money and use fewer resources. 

Solution: In conjunction with the CTL and Professor Nemrow, The School of Accountancy developed a 
value-added model to teach the course more effectively and to save money. The primary component in this model is 
the Introductory Accounting: The Language of Business™ CD. The CD lessons are taught in an audio-video 
lecture-style format. Each CD contains video of a professor lecturing with vector-based graphics appearing to the 
side of the video as additional instruction material. 

Results: This approach has improved cost effectiveness in several ways. First, the CD-based course saves 
substantial professor time. Students are able to review materials and drill down into additional remedial content at 
their own pace on their own time. This makes class time significantly more effective. When students come to class, 
instead of using valuable class time to review material, they take the principles that they learned from the CD and 
apply them to real-world problems and cases. Using the CDs also saves the students money. For example, the six- 
disc set of accounting CDs for the course is sold at the campus bookstore for $65. A new accounting textbook costs 
approximately $130. Student interest in accounting as a major has doubled since the creation of the CDs. 
Approximately 90 percent of students who take this course prefer the CD-based approach, rather than the traditional 
lecture/textbook format. 

Example 2: Blackboard Quiz Feature 

Problem: BYU uses Blackboard as its enterprise course management tool. With almost 50,000 users 
(instructors and students). Blackboard is not inexpensive to maintain and support. But it provides substantial cost 
effectiveness improvements for faculty and students. One significant problem it helps ameliorate is the amount of 
time required administration of formative assessment (quizzes) in the classroom. 

Solution: By providing automatically graded, low-stakes assessment online via Blackboard, faculty at BYU 
have been able to provide substantial formative assessment and feedback opportunities for students without taking 
away from class time. 

Results: System logs and survey response data suggest that BYU students and faculty are using the 
Blackboard quiz feature extensively to facilitate a valuable learning function in a very cost effective way. On 
average, faculty deploy more than 30,000 assessments a year which are completed by students and graded by 
Blackboard about 4.5 million times (approximately 2 million “events” per semester). In a survey taken of 238 BYU 
students, 224 had taken a quiz in Blackboard for at least one of their courses — 67% had taken Blackboard quizzes in 
at least four courses. Eighty-one percent of these students agreed or strongly agreed that the Blackboard quiz 
features saved them time in their courses. Approximately 50% of respondents (226 students) strongly agreed or 
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agreed that the Blackboard quiz feature improved their own learning in their courses. 58% of faculty (97 of 166 
faculty) surveyed said they have used the Blackboard quiz feature in their courses. An impressive 62% of these (61 
of the 99) agreed or strongly agreed that Blackboard quizzes improved student learning. 

Access Examples 



Example 1 : Virtual Audiometer 

Problem: Prior to the development of the Virtual Audiometer, a professor at BYU wanted to show his 
students how to properly conduct a hearing test. To demonstrate this task, he would set up video cameras and use a 
portable audiometer. However, there were many students in the class and these students had difficulties seeing what 
the professor was doing and also had problems discerning which buttons he was pushing when conducting the test. 
Eor homework assignments on the task of using an audiometer, the professor would group the students together and 
have them use the portable units. The benefit was that students received some hands-on experience using the 
equipment. Unfortunately, simulating authentic hearing impairments was very difficult since the students were 
testing each other. 

Solution: To address these limitations, the Department of Communication Disorders partnered with the 
CTL to develop the Virtual Audiometer. This product was designed to help students learn audiometric testing faster 
and more efficiently than by using the portable audiometer in a classroom setting. This highly interactive software 
recreates the audiometric testing process in remarkable detail, dramatically reducing learning time without 
compromising the value of live patient-to-practitioner interaction. Using the Virtual Audiometer, students have 
access to thousands of customizable virtual patients, providing them a wide range of opportunities to practice their 
audiometric testing skills. Eaculty can also design virtual patients to provide specific teaching and testing 
opportunities and to give meaningful, individual feedback. Using the Virtual Audiometer, students can practice their 
skills anytime of the day on virtual children, adults, and elderly patients without the constraints of a limited clinical 
space. Eaculty can also use the Virtual Audiometer for in-class for demonstration of procedures. Essentially, the 
Virtual Audiometer provides many of the benefits of live patients, without the prohibitive cost or time investment. 

Results: The Virtual Audiometer is used nearly every day in audiology and speech language pathology 
classrooms. Students who have used the Virtual Audiometer were very satisfied and also reported a good learning 
experience. The reasons students like the Virtual Audiometer are as follows: (a) realistic simulation (29 percent), 

(b) convenience, repeated practice and good use of technology (26 percent), and (c) experienced virtual patients (25 
percent). In another study conducted by Johnson, Graham, and Su-Ling Hsueh (2006), they discovered five 
additional benefits of using the Virtual Audiometer, which include the following: (a) visualization, (b) authentic 
engagement, (c) quality and quantity of practice and feedback, (d) interaction and collaboration, and (e) reflection. 
Eirst, they found the Virtual Audiometer helped students visualize content better. Essentially, they could see very 
clearly what the professor was doing when he was using the Virtual Audiometer. For example, one student said, 

“We were able to see what buttons he was pushing and what frequencies he was testing. . .a wonderful way to 
present to a class this size.” In terms of authentic engagement, students felt that using the Virtual Audiometer helped 
them gain a greater understanding of the concepts of hearing loss. 

Example 2: Independent Study Courses 

Problem: A variety of factors such as family commitments, job, and location can make it difficult for 
students to attend classes on campus. Further, some students want to graduate early from high school or simply need 
a few extra credits to graduate from college. Other students cannot attend on-campus courses because of illnesses 
and other challenges. Still, there are some students who are simply looking for increased flexibility and the 
opportunity to take courses “however, wherever, and whenever” they want. 

Solution: BYU’s Independent Study program helps meet the needs of these students who want educational 
alternatives. Their model is “education for anyone, anywhere, anytime.” BYU Independent Study currently offers 
more than 600 university, high-school level, and personal enrichment courses. Students also have access to an online 
tech support team, as well as a help line. BYU’s Independent Study program is the largest university-run distance 
education operation in the United States. 
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Results: BYU’s Independent Study program is fully accredited. Students from all 50 states and 63 foreign 
countries have enrolled in these courses. Course enrollment totals more than 150,000 a year. Students are satisfied 
with the courses because they can work at their own pace, receive input from professors on their work, take 
advanced or specialization courses, and have the flexibility to work in the comfort of their home, or wherever they 
may be. Upward trends in enrollments, re-enrollment by students in additional courses, and overall student 
satisfaction rates indicate that Independent Study is effectively achieving its objective of providing meaningful 
educational experiences to students who would otherwise not have access to them. 

Student Satisfaction Examples 



Example 1: Histology: 

Problem: Histology — the science of tissues and their cellular structure — is best understood by seeing 
examples. Historically, the study of human histology has involved the preparation and detailed examination of thin 
sections of human tissues that have been fixed, stained, and mounted on glass microscope slides. Histology students 
would typically spend many hours studying hundreds of slides on the stage of the microscope in order to learn the 
salient characteristics of different tissue types. This study process contains several limitations. Eirst, the student must 
have access to a collection of slides and a microscope. Second, the different cellular features of the images on the 
slides are usually not labeled, so a fair degree of interpretation is required to determine their identity. 

Solution: To help in this tedious learning process, the CTL designed an interactive histology encyclopedia 
that illustrates the most significant types of human tissues and their diagnostic characteristics. This encyclopedia 
allows students to gain experience in histology using glossaries, quizzes, and microscope slides. The encyclopedia 
uses thousands of actual light and electron microscope images in a digital format. The tutorial provides dynamic 
feedback to the user, and also has a self-check and exam feature to allow students to test their knowledge of human 
tissue types. The encyclopedia also serves as a study aid to supplement course text and lectures, and is available 
through the Internet and CD-ROM. 

Results: The Histology encyclopedia is a valuable aid for both students and instructors as it reduces 
laboratory and classroom time spent in learning routine identification skills. Many students who used the 
encyclopedia said it is easy and enjoyable to use and that their learning increased from using the tutorial. Some even 
commented that the encyclopedia was vital to their success in the course and that it was more effective than the 
textbook. The students also enjoyed being able to study on their own time. This tutorial also contributes greatly to 
the pillars of access and student satisfaction. Access is increased as students now have the ability to view high- 
quality digital histological images on their personal computers at any time and in any order that they need, and can 
replay the materials as often as necessary to teach themselves how to recognize the various samples of human tissue 
types. 

Example 2: Online Student Ratings 

Problem: When students register for school, they often talk with friends and roommates about which 
classes they should take and from whom they should take these classes. Of students surveyed recently at BYU, 77% 
reported that they use RateMyProfessors.com to glean information regarding who should be their teachers. 
Unfortunately, there are some drawbacks to using RateMyProfessor.com, such as: (a) no restrictions on who can 
respond, (b) responses may not be updated to reflect the current course or most recent semester, (c) no system is in 
place to ensure reliability and validity, and (d) students who post ratings often have extreme views. 

Solution: BYU has been soliciting student ratings of instructors and courses online since Pall Semester 
2002. Given the advent of sites like RateMyProfessor.com and their drawbacks, BYU administrators wanted to 
provide students with accurate information about faculty performance in the areas of teaching and learning. They 
wanted students to be able to receive accurate information and learn as much as possible from the ratings instead of 
relying on often incomplete and inaccurate information. After careful consideration (including additional research 
on teaching, learning, and student-ratings, and conducting several campus pilots), BYU decided to make responses 
to the core questions from its online student ratings available to students. 
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Results: Any BYU student who fills out the online student ratings can view all professors’ ratings the next 
semester in the following areas: (a) amount learned, (b) materials and activities, (c) intellectual skills developed, and 
(d) interest in student learning. Students are satisfied with the online student rating report because they receive 
reliable and accurate information from students who are currently enrolled in the course. Since BYU just provide 
partial access to student ratings in May 2008, we have only limited evidence of student satisfaction with this 
initiative. However, BYU hopes that this initiative will increase student response rates and provide students with an 
excellent source of accurate information about professors and courses. We are encouraged by the results of similar 
efforts at other institutions. For example, in 2004 Northwestern University began requiring students to complete 
online student ratings of their own courses in order to view other student ratings results. This change increased 
student response rates by 25 percent. We are also focusing significant attention on real improvement in teaching and 
learning, as evidenced by improved student ratings. The vast literature (exceeding 1500 articles) on student ratings 
and student learning demonstrates high correlations between the amount students learn and their overall rating of 
those courses. 



Faculty Satisfaction Examples 

Example 1: Media Notes (formerly know as Performance Analysis Tool [PAT]) 

Problem: Two law professors at BYU teach courses on counseling, interviewing and negotiation skills. 
These professors felt that their students were learning these skills adequately, but that there was still room for 
improvement. In the past, these professors would have their students practice these skills live in front of the faculty. 
The students were required to prepare for the performance, perform a practice exercise, and then reflect on their 
performance. Part of the reflection always included a self-evaluation activity, which both professors felt was 
important. Unfortunately, there were several problems associated with the self reflection. Eirst, after the 
performance, students had to rely on their memories to reflect on their 45-minute performance. Second, if students 
wanted to have notes on their performance, they would have to keep notes synchronously with their performance. 
Third, without recording the performances, it was very difficult for the professors to evaluate the performances of 
their students and provide precise feedback to them. 

Solution: To help alleviate this problem, the professors turned to using videos in 2004. Unfortunately, there 
was not enough equipment for all of the students (typically about 40-50 students per class), and the process was very 
time consuming. Eurther, capturing the video also required the assistance of Media Services at BYU, further 
complicating and limiting the flexibility of faculty members to conveniently capture, review and provide feedback 
on student performances. All of this took a great deal of time, time which they began to feel was excessive and even 
wasteful. Recognizing the importance of repetition and quality feedback in helping students learn negotiation and 
interviewing skills, they wanted to find a more time-effective way to review student performances and provide 
meaningful feedback. The negotiation faculty members and the Law School partnered with the CTL to implement an 
approach involving webcams and a BYU-developed tool now called MediaNotes (formerly called Performance 
Analysis Tool [PAT]). 

Results: Using MediaNotes, students can easily record their negotiations, and use performance evaluation 
templates to reflect on their experience, identify problems that need to be corrected in the next practice, comment on 
specific parts where they did well or would like feedback, and then upload the file to a server on campus. Tagging, 
categorizing and attaching written comments within the video recording are simple, straightforward tasks in 
MediaNotes. Eaculty can view the videos, quickly view segments where students want feedback, record feedback, 
and upload a text file (not an edited video file) to the server. The students can use MediaNotes to quickly see where 
the professor provided comments without listening to the entire recording. Students also benefit because they have a 
copy of the performance they can watch without having to rely on their memories. Eaculty members are satisfied 
because they are able to provide meaningful practice opportunities and targeted feedback while spending much less 
time than they did before using MediaNotes. Even more importantly, the faculty members are satisfied because 
students are more effectively mastering negotiation and interviewing skills. As the students have used MediaNotes, 
the professors have been able to refocus some of their course content, reducing the amount of time needed to cover 
some theoretical material and also decreasing time spent on teaching methods. Using deliberate practice methods in 
conjunction with the MediaNotes software has significantly changed course and classroom dynamics and has even 
increased student participation and motivation. Learning effectiveness has also improved. Eor example, comparing 
student performances from five years ago with performances done by students using MediaNotes shows that 
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students using MediaNotes are considerably better. Research still needs to be conducted on knowledge and skill 
retention and transfer. Student satisfaction has also increased as students have greatly enhanced flexibility in their 
review of their performances. 

Example 2: In-Class Response System (iClickers) 

Problem: Many faculty members have a desire to make their classrooms a haven of learning in which 
activities are effective and useful. Oftentimes, however, faculty members are left unsure of their effectiveness and 
usefulness. After a lecture or a class discussion, it is difficult to know how well the students understand the material. 
They may call on students to respond to their questions; however, usually only a few students have the opportunity 
to participate. If faculty members administer a quiz or test, they usually have to take time outside of class to grade 
these papers and evaluate how the students are doing. Rarely do faculty members know instantly and with certainty 
how well their students have learned a particular concept that was just taught. This knowledge gap often leaves 
instructors feeling frustrated and unsatisfied with their efforts. 

Solution: To close this gap, an In-Class Response System (provided by iClickerj has been implemented in 
many classrooms at BYU. Using the iClickers, students can immediately respond to questions that are then 
displayed for immediate discussion and analysis. Faculty members receive immediate feedback on how well their 
students comprehend the material. Once the students respond using the iClickers, the responses can be immediately 
processed and projected onto the screen in a form of a graph to be stored for review or for analysis. Faculty no 
longer have to wait for students to fill out a paper quiz and then correct it to obtain information. They can also easily 
obtain feedback from all of the students, rather than from just the outspoken students in class. Faculty can then direct 
their class discussions and lessons based on the feedback they receive and the needs of the students. 

Results: The iClickers are flexible, interactive, and maximize student participation, particularly in large 
class environments. They can be used for quizzes, tests, roll call, gathering data, to initiate discussion, or even brief 
find-out questions to gauge understanding during class. Virtually every faculty member who has used the iClickers 
has been satisfied with their use. In essence, the iClickers save the faculty members time and energy, allow for 
interactivity between the teacher and student, and provide rapid feedback. Other benefits for using the iClickers 
include improved class attendance and preparation, increased participation during class, and increased retention. The 
frustration of not knowing how well students are progressing during a class session is largely eliminated. Students 
also enjoy using the iClickers because they can engage in an interactive conversation with their instructor and 
classmates. Shy students can also use the iClickers to register their opinions on topics in a non-threatening 
environment without having to raise their hands and speak to all of the class. 

Conclusion 

In this paper we have provided ten examples of projects that were completed at the CTL using a design- 
based research approach based on the Sloan Consortium’s (Sloan-C’s) Five Pillars of Quality Online Education. 
These and other similarly goal-driven projects demonstrate that technology enhancements can and do make a 
significant difference in terms of increased learning effectiveness, cost effectiveness and institutional commitment, 
access, faculty satisfaction, and student satisfaction. 

While we did not formally include a Sloan Pillars analysis step in every project summarized above 
(particularly the less recent projects), we have gradually internalized and more formally adopted principles of the 
five pillars into all of the projects we discussed in this paper and others we have completed at CTL over the past 
decade. Moving forward, we now consider each project’s goals and their relation to the pillars as a formal step in 
our design process We are confident that by consistently implementing this step in each new project we will 
continue to see and be able to demonstrate that instructional technology improvements do, in fact, make a significant 
difference for teaching and learning at BYU. 

Future research should focus on the relative value derived (i.e. the amount of teaching and learning 
improvement) from different levels of investment (i.e. resources required to implement various technological 
enhancements). Researchers should also focus on refinements to the goal articulation and strategy formulation 
processes. Alignment between goals, strategies and tactics and the evaluation process are at the heart of the design- 
based research process and should receive heightened attention from researchers and practitioners. 
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For now, we are content to conclude that our approach to project design and development using 
technology, significant differences can be achieved and demonstrated in teaching and learning. No longer should we 
reach the conclusion of “No Significant Difference” when using technology to enhance teaching and learning. Either 
our strategies and tactics help us achieve stated teaching and learning goals or they do not. As we continue to 
implement and refine our approach to instructional design and evaluation, we recognize the need to educate and 
interact with administrators, faculty and students to fully integrate an effective process of continuous teaching & 
learning improvement and feedback based on a design-based research approach, rooted in the five Sloan pillars. 
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Abstract 

The purpose of this paper is to share effective strategies in the training of teachers and teacher candidates in 
integrating technology and the tools of Web 2.0 in diverse classrooms when limited by budget, infrastructure, and 
shelf life. This paper will be of interest to teachers, teacher educators, media specialists, and administrators whose 
goal is to prepare students for the global economy by exposing them to new/emerging technologies regardless of 
budgetary constraints and obsolescence. 

Globalization catapulted Information and Communication Technologies (ICT) planning in education 
universally. Government and educational leaders collectively recognized the crucial need to create “work ready” 
citizens who can compete in the global economy; which at a minimum necessitates literacy, math and science skills, 
and competencies in new and emerging technologies. Ostensibly training begins in educational institutions. 

However the reality is that a majority of our public institutions, K-12, and Institutions of Higher Education (IHEs) 
are constrained by non-existent or limited budgets, outdated infrastructures, and old software. Inexplicably, the 2007 
federal budget proposal once again called for additional cuts in education, including the elimination of No Child 
Left Behind’s (2001) block grant program of $275 million aimed at Enhancing Education Through Technology 
(LETT). Although we are in our fourth wave of ICT planning, many issues remain unresolved. 

The purpose of this paper is to share effective strategies in the training of teachers and teacher candidates in 
integrating technology and the tools of Web 2.0 in their diverse classrooms when limited by budget, infrastructure, 
and shelf life. Recognizing the arrival of the millennials, many of whom who are adept in digital technologies due to 
resources provided in the “home” environment (Visions, 2020), our short-term goal is to provide equity in 
technology resources for all citizens. Our long-term goals are to have teachers and teacher candidates prepare their 
own students for the global economy and “turnkey” the instructional strategies in using free web sites and open 
source software in their classrooms (as there will always be a digital divide.) 

Course Management Systems 

Although diversified strategies will be shared under the following college course nomenclatures, the 
underlying principals remain the same: students are exposed to free resources and open source software that are 
current and emerging technologies and are available to all regardless of socio-economic status. A mainstay of both 
authors’ strategies is that students harness the power of the Internet and utilize the interactive tools of Web 2.0 that 
can be accessed in educational environments and non-profit institutions such as local libraries and universities. In 
addition, both professors utilize the course management system, Blackboard, as it is provided by the college; 
however, both professors could alternatively use the free course management system Moodle. 

Open Source Software 

In the graduate level course. Advanced Pedagogy and Curriculum II, teachers and teacher candidates, in 
techno-savvy pairs (a novice user paired with an experienced user), create in the constructivist approach a six-lesson 
intermediate interdisciplinary science inquiry unit that adheres to Bloom’s Taxonomy (1956), addresses each of 
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Gardner’s Multiple Intelligences (1993), and incorporates mathematics and new and emerging technologies (see 
Figure 1). Utilizing free resources on the web, students become well- versed in the content and process standards set 
by the National Council of Teachers of Mathematics (2000-2004), Standard 1 (Analysis, Inquiry, and Design) and 
Standard 4 (The Physical Setting and The Living Environment) of the Elementary Science Core Curriculum Grades 
K-4 (n.d.), and the National Technology Standards for Students (2007) and Teachers (2008) and their behavioral 
objectives reflect these benchmarks. Students are initially exposed to diversified resources which are housed on 
Blackboard and open source software in word processing, graphic organizers, and multimedia tools. Several 
programs are downloaded such as OpenOffice.org (1995-2007) and AbiWord (1998-2007), which emphasize word 
processing and spreadsheets, and are compatible with the Microsoft Office Suite. Students are also encouraged to 
work in techno-savvy pairs (O’Connor-Petruso, 2003a; 2003b), instructed to follow one of the many free internet 
tutorials on the web, and if needed, given “hands-on” instruction by a colleague and/or the professor. 

Web Pages 

Students also create their first web page on a particular elementary science inquiry unit through the free 
template Eilamentality (2007). Eilamentality is an AT&T and government sponsored web site, offering free web 
hosting and a plethora of diversified templates that encourage students to create web pages. In order to generate 
maximum visibility, provide examples of exemplary student created web sites, and motivate future web authoring, 
their completed Eilamentilty sites are posted by the professor on http://www.wix.com. Although the actual URL 
denoting the site is extremely lengthy, students have an online publication on the professor’s site. They also have 
increased their web resources, can integrate theses tools into the classroom, and turnkey the strategies they learned 
in their college classroom to their students who may be motivated to create their own Eilamentality sites. 

Recording Tools 

Students are also exposed to the open source web recording tools such as CamStudio (2007), free streaming 
video software, and Audacity (n.d.), open source software for recording and editing sounds available on multiple 
platforms. Similarly, if students own MP3 players they can incorporate their own podcasts into their multimedia 
presentation. 

Blogging 

Students are also given the option to discuss concerns and issues on Blackboard’s Discussion Board or on a 
self-created Math/Science/Technology Blog - their choice as free blogging exists on numerable sites. Although the 
professor does not monitor the blog, net-etiquette and legal ramifications of blogging are discussed in detail. In 
addition, teachers and teacher candidates are also exposed to the Education Policy Blog (2007) that discusses 
National Council Accreditation of Teacher Education (NCATE) Professional Dispositions. 

Web Hosting 

Technologically savvy students also create personal index pages on their choice of free web hosting sites. 
Touted for user-friendliness and multiple attributes such as widgets and blogs, http://tripod.com (2008) and 
http://www.Wix.com (2006-2008) offer users the opportunity to upload pictures and animation, insert sound, and 
create professional looking websites in a matter of minutes. Students are exposed to the various tenets of several 
wiki tools and are given instruction on how to access and Join the professor’s created web site 
http://MyMstWiki.wikispaces.com, view pages, paste text, upload files, save files, and view history in case they 
accidentally delete part or all of the pages - which they do! 

Wikis, Blackboard, and Other Ereeware 

The culminating project in this Advanced Pedagogy II course is a multimedia presentation on the highlights 
of their science inquiry unit. The presentation underscores their top behavioral objectives and Web 2.0 tools they 
have incorporated in their lessons, provides examples of student target based assessments and rubrics, and is 
uploaded on the wiki. Again the underlying goal for posting these assignments on the wiki is to motivate these 
teachers and teacher candidates to continue to explore and integrate Web 2.0 tools in their classroom and most 
importantly - turnkey these strategies to their students who will become technologically savvy, motivated to become 
life long learners in the chameleonic realm of technology (O’Connor-Petruso, 2004) and thus be competitively 
prepared for the global community. 

In the two semester graduate level Action Research courses, teachers and teacher candidates are again 
exposed to Blackboard, Wikis, and freeware. Blackboard again is utilized for accessing course syllabi, assignments, 
resources, external links for accessing open source resources and statistical software, and Discussion Board is used 
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for online assistance from colleagues and/or the professor. In preparation for their final Action Research Project, all 
in-class assignments are posted on the wikis that the professor reviews in class. Students overwhelmingly tout the 
need for visuals as they provide examples for those who need assistance with the assignments. In the initial research 
course, students post their work on http;//Early ActionResearch.wikispaces.com. During the second semester 
students view their prior colleagues’ completed Action Research Projects on http://ActionResearch.wikispaces.com 
and eventually post their final Action Research Projects. 

Virtual Field Trips 

In another graduate level course, teachers find information on a web site to extend or enhance a lesson. 
Paying special attention to national, state, and curriculum standards, students develop a virtual field trip (VFT) 
lesson plan. Although a real field trip is generally viewed as an excellent way for students to have a firsthand 
learning experience (Smaldino, Fowther, & Russell, 2008), in areas where distance makes such a trip impossible or 
in places where funding is an obstacle, a VFT is a way to give all students an opportunity to “visit” a place outside 
of school. VFTs can be custom designed by a teacher or they can be found “ready made” on web sites. How a VFT 
is used in the classroom depends on the teacher’s learning objectives and availability of computers per student. 

VFTs can be teacher-led or student-centered. Students may work individually or in cooperative groups. The 
variation in the way the VFT is conducted makes it fairly easy to incorporate into the instructional unit. Although 
VFTs are not necessarily Web 2.0 tools, they are, nonetheless, freely accessible. 

Film Making 

If a teacher wishes to approach a curriculum area in a different way, capturing a story on film can be a 
wonderful change of pace. Many content areas can be addressed, for example, teaching a difficult math concept to 
others, enacting or making changes to a favorite story, filming a class play so that family members can see it later, 
recording a science experiment, or interviewing elderly people in the community to find out what life was like fifty 
years ago. If a class has access to a digital camera, there are some free applications for film making, such as Movie 
Maker, and for creating slides, such as Photo Story. Using a camera to tell a story is another way for students to 
express themselves as they learn. There are many roles in which students can be involved in a film making project, 
including those dealing with directing, camera operation, scenery and props, lighting, scripting, and acting. Students 
can collaborate on this type of project, making the learning interpersonal as well as content driven. 

YouTube 

YouTube is a relatively new phenomenon that can also be employed as an engaging learning tool. Many 
videos have been posted that can be useful for student learning. For example, if a teacher was using oobleck to 
discuss the properties of matter, there is a funny video (in Spanish - although the language is of no consequence) 
showing two fellows running across a pool of oobleck without falling in 

(http://www. youtube. com/watch?v=yHlAcASsf6U). This could be a very interesting motivational tool for a 
conversation about the properties of oobleck. 

Cell Phones 

The use of cell phones appears to be ubiquitous, even in low-income areas where many students own cell 
phones (Manzo, 2008). Although most school districts in the United States ask students to turn cell phones off in the 
classroom, in New York City, cell phone use has been banned in the classroom because it is thought to be a 
distraction to learning (Gewertz, 2006; Vaishali, 2007). Sometimes cell phones are disruptive, for example, when 
they ring in the classroom. Students abuse them when they use the camera function to copy tests or answers. 
However, forward-looking educators are suggesting that the multi-function cell phone could also be a good teaching 
tool in the classroom. Manzo reports on some suggested uses of a cell phone that include recording first-person 
interviews that can later be posted on wikis, Web sites, or somewhere else on the Internet to augment a report or 
project. Students can record themselves playing an instrument or speaking another language and they can then send 
these audio clips to their instructors. In the classroom, the cell phone can be used to quickly poll student opinions. 
Teachers can make videos of lesson plans for a substitute teacher and send them via cell phone. When parents and 
teachers have difficulty communicating because of a language problem, messages can be translated and then 
transmitted by cell phone. As new technologies become available, educators need to think about ways to harness 
their potential for classroom use. 
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Conclusion 

We tend to teach the way we are taught, what Lottie (2002) referred to as an “apprenticeship of 
observation.” Some teachers have been fortunate to have had technology experiences in their classrooms as students. 
However, often teachers have not had experiences teaching with technology, even if they have seen it modeled 
during their student years. Teachers need practice with new tools to become comfortable and confident users before 
they introduce these tools to their own students. Using Web 2.0 and other freely accessible technology tools can help 
to broaden the technology abilities of all teachers who can then be better prepared to help their own students in the 
learning process. 

In an economically downward market, in which the American dollar continually loses leverage worldwide, 
it is critical that professors expose teachers and teacher candidates to the new and emerging free tools of Web 2.0 as 
the global economy necessitates these skills. Similarly, the majority of our students cannot afford to purchase new 
technologies and school districts overall may have outdated software and infrastructures. One out of every three 
American students (Burr, 2008), approximately 1 .2 million students annually, drop out of high school and the 
twelfth grade literary scores (NCES, 2007) are the lowest in twenty years. Perhaps these new free and emerging 
tools of technology, if tumkeyed properly to the K-12 students by technologically savvy educators, will provide 
intrinsic motivation for students to learn Information Age skills and want to become productive and competitive 
citizens (Dewey, 1916) in the global age. 



FIGURE 1 

MST INQUIRY UNIT MODEL 

O'Connor-Petruso, 2001; 2005 




Additional Performance Based Assessments of the MST Inquin' Unit Model 

1) Children’s Literature for each lesson: Text, Magazines, Newspapers, Web Resources 

2) Two Manipulative Skills from the Program Evaluation Test (PET) 

3) Three Mathematical Representations including congruent spreadsheets and'or charts: Bar Graph, 
Line Graph, Pie and'or Picto-Graph 

4) One Scavenger Hunt 

5) One Graphic Organizer 

6) One FilamentaliW Website 

7) Minimum of One Web 2.0 Tool: Blog. Wiki, and/or Podcast 
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A Design and Research Framework for Learning Experience 



Patrick Parrish 
The COMET Program 
Brent G. Wilson 
University of Colorado Denver 

Recent literature in a variety of design fields has called for a shift in focus from products and usability, and 
from effectiveness and efficiency, toward characteristics of user-experience (McCarthy & Wright, 2004; Hassenzahl 
& Tractinsky, 2006). For example, instead of simply making products that work and provide ease of use, designers 
are looking more at ways to provide meaningful and enjoyable use - culminating in a satisfying and fulfilling user 
experience. While creating a technically reliable, easy-to-use product clears an initial hurdle, many designers are 
looking for ways to deepen and strengthen the user’s experience with the product. 

This shift from utility to user experience is not purely altruistic of course - it is largely a response to 
increasingly competitive market conditions. Users of educational products, for example, face a glut of messages 
demanding their attention about an intimidating array of products and choices. Within an “attention economy,” 
learners attend to messages that are inviting and engaging in addition to those expected to fill functional needs (de 
Castell & Jenson, 2004). For good reason instructional designers have shown increasing interest in exploring the 
learning experience, frequently drawing from the sister design fields and from the arts (Dickey, 2005; Hokanson, 
Miller, & Hooper, 2008; McLellan, 2002; Parrish, in press; Wilson, Parrish, & Veletsianos, 2008). The surge of 
interest in educational gaming reflects this new focus on learner experience. 

A similar shift is happening in the health professions, with new emphases on patient wellness (not just a 
physical state, but a relationship of increased connectedness to one’s body and the world) rather than merely curing 
illness, and on nursing care and presence (being there for all patient needs, including social, emotional, and spiritual 
ones, rather than just carrying out patient treatments — see Watson, 2008; Woodward, 2003). This is partly a 
competitive response, based on perceived threats to the profession from alternative medicine and standardized 
healthcare delivery. But it originates in the recognition that traditional approaches are limited in their impacts, often 
unnecessarily costly, and, in their focus on technical solutions, frequently fail to address quality-of-life issues. 
Instructional designers might learn from the health professions through a similar broadening of intended outcomes, 
substituting wellness with its educational parallels, including open-mindedness and a healthy desire to learn. 

A focus on the learning experience raises many questions for instructional design (ID) practice. Unlike 
behavioral objectives or discrete cognitive skills, the construct of learning experience lacks the precision or control 
that instructional designers are accustomed to. The notion of experience is more holistic, requiring simultaneous 
attention to cognition, behavior, and affect - even agency and identity. Clearly a conceptual framework of the 
learning experience is needed. This could then serve as a basis for advancing knowledge through research and 
theory, and shaping the learning experience in real situations, as instructional designers would be called to do. 

This paper offers a framework for understanding the learning experience that can guide instructional 
designers in creating more engaging designs and contribute toward a research agenda for advancing knowledge in 
this area. The framework is consistent with pragmatist and phenomenological perspectives of experience, and is also 
informed by current learning theory. 



A Range of Learning Experiences 

Before detailing the framework, the next section describes two contrasting cases. They diverge in several 
dimensions that will become clearer as the framework is discussed. We will revisit them later when we apply the 
framework to analyze their qualities. 

Pat’s Experience with Psychology 302 

As an undergraduate, I (Pat) took a sophomore-level course in Developmental Psychology to fulfill one of 
my core requirements. This was a popular course, and so it was taught in a large lecture hall to more than 200 
students. As was my habit in such courses, I sat nearer the back than the front of the hall, and rarely offered to 
interact by asking questions or providing answers. Instructor contact consisted of twice-weekly lectures, following 
the comprehensive text students were assigned to read in sequence and nearly in its entirety. Lectures were 
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interesting because the content was interesting, and the instructor was personable, but beyond using infrequent 
photographic slides and reporting on additional research, the instructor did little to expand the content beyond the 
text. The text was organized chronologically by developmental stages, and interwove theoretical explanations as 
they applied to those stages. It was easily recognizable as a standard college textbook, using a clear but 
unimaginative writing style and predictable end-of-chapter comprehension questions. Assessments included three 
long exams composed of multiple-choice and short response items, administered by a teaching assistant. 

Because I had not yet taken advantage of my university’s policy to offer students the option of taking up to 
two core course on a pass/fail basis, I did so in this case, assuming the reduced pressure would allow me to put more 
effort into my major area literature and film history courses. While the course content interested me, I found myself 
putting in minimal effort. In fact, having passed the two mid-term exams with high grades, and knowing that the 
lowest test grade would be thrown out, I naively decided to skip the final exam, assuming that failing grade would 
be thrown out. The teaching assistant thought “it probably would be ok.” It wasn’t, and I received a grade of 
incomplete for the course. 

That summer, I was required to meet with the course professor — the first time I’d seen her at a distance less 
than 25 feet — and she assigned an alternative final exam composed of short essay questions. I enjoyed the brief 
meeting, and in many ways, I found the final exam interesting to complete. While I never saw a final grade. I’m sure 
I did well. 

All in all, I did not consider it a bad course at the time, but as might be expected, I retained only minimal 
knowledge of the topic from the course. Most of what I have learned about developmental psychology has come 
from later readings done on my own and for other courses. 

Erin’s Arctic Adventure 

This fictionalized report is based on descriptions of Arctic Transect 2004 (Doering, 2006), an online 
“Adventure Learning” program offered to K-12 teachers for use in their classrooms. Erin was an 8* grade student 
when she participated in Arctic Transit 2004. Her science and humanities teachers decided to collaborate in 
employing the program to enhance their standard curriculum during spring semester. Arctic Transit was centered on 
an adaptable, problem-based curriculum that could be downloaded on demand from the program Website. The 
curriculum was linked to a real-time, 3,000-mile dogsled expedition by educators and adventurers through the 
northernmost Canadian province of Nunavut. Erin’s teachers chose to implement units that challenged students in 
collaborative problem-solving activities (with students from around the world) to understand Inuit culture, language, 
and, in particular, their ecological knowledge and strategies for adapting to their often harsh environment, and the 
potential impacts of climate change. 

The science and humanities text-based units were augmented by classroom and Internet-based activities, 
and live and recorded reports from the expedition participants as they travelled through the region interacting with 
the Inuit people. The Internet activities included chat sessions with expeditioners and experts, during which Erin 
submitted questions that were directly or indirectly answered. Erin also viewed online photos and videos, and 
downloaded audio reports from the expedition that expanded upon the text. Classroom activities asked Erin to 
connect the course content to her own life, such as her own family’s environmental impacts and cultural practices. 
She shared reports of these online with the worldwide cohort of fellow students. But her favorite part of the program 
was learning about the team of dogs pulling the expedition sleds, and following the adventure from their perspective. 
She checked in weekly for an update “written” by one of the dogs. A portion of the Website focused on the dog 
teams also allowed her to share photos and stories of her own dog. 

Erin often found herself having so much fun with the online collaborations and opportunities to share her 
insights and experiences that she put in more effort than she usually did for school work. She looked forward to the 
upcoming chat sessions and the weekly reports from the field, and always found time to read the assignments that 
prepared her to participate. The experience has increased her interest in geography and environmental science to the 
level that she has read additional books and articles on similar topics. She remembers it as one of the most fun times 
she had learning in middle school, and hopes she can convince her high school teachers to participate in new 
adventure learning programs. 
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A Framework of Experience 



Experience is partly an internal process, but it is more than the collection of psychological states undergone 
by an individual in a given situation. By the same token, experience contains external conditions and events, but it is 
more than merely something that happens to a person. In other words, it is neither merely an individual’s subjective 
responses to a situation nor just the objective conditions that make up that situation - and it is not merely the 
addition of the two. It is useful to view it more broadly as the transaction or engagement that takes place between 
an individual and the world (Dewey, 1925/2000). Experience in this sense is a reciprocal activity that includes a 
conscious individual engaging with a responsive world — a world that includes objects, physical qualities and other 
people. The individual and the world are co-creators of the experience. Erom the transactional point of view, the 
value of an experience can be described in terms of the quality of the transaction that develops and its potential to 
improve future experience (Dewey, 1938/1997). In turn, the qualities of the experience worth exploring are those 
that impact the nature of the transaction. 

Although not fully captured by a breakdown of internal and external elements, experience includes: 

• A person’s active engagement in an activity involving the outside world, including material 
objects and other people 

• A person’s immediate apprehension of that ongoing activity, including the physical/bodily 
response to the situation 

• A person’s construction of meaning from that activity, typically built around a narrative 

• The intersubjective, joint construction of meaning of groups of participants 

• The responses and changing nature of the world that dynamically impacts a given situation 
Thus when we ask a learner what meaning they derived from an activity, we are probing experience - but 

experience is not reduced to their report or their construction of meaning. Maintaining a holistic, transactional view 
of experience helps avoid reducing the construct to a psychological state or process and maintains the unit of 
analysis at the person + world level. 

The instructional model of experiential learning (Kolb, 1984) is well documented and has been adopted by 
many practitioners. While we view it as strong model for developing powerful learning experiences, the range of 
potential qualities of experience are not addressed by this primarily prescriptive model. We share the belief with 
Kolb that all learning happens within experience, but are unwilling to call some forms of learning "experiential" and 
others not. Our goal here is to account for the continuum of learning experiences by identifying the qualities that 
lead to those with the most powerful learning potential, but not to prescribe a particular sequence of instructional 
events. 

Interest in the construct of engagement has increased in recent years (e.g., Dickey, 2005; Greeno, 2006; 
Hung & Khine, 2007), partly to acknowledge its key role in flow-like activities such as game playing, and partly out 
of dissatisfaction with constraining models of motivation available from cognitive psychology. To some extent, 
engagement with the world leads to experience. In another sense the two terms are synonymous — they both refer to 
activity with the world. Engagement both leads to and is part of a learner’s experience with the world. In general, 
engagement becomes an important part of our language when trying to articulate the processes of experience. 

Temporal Dimensions of Experience 

Experience has several temporal dimensions that account for its nature and potential value. 

• It is immediate. Experience is felt, not just observed or reflected upon. An individual’s relationship to the 
situation at a given moment, before rational analysis and when affective influences hold at least equal sway 
to cognition, is a critical factor in the ultimate value attached to it. The qualities of immediate experience 
can color all other aspects, determining how deeply one engages and the meaning one attaches to it. 

• Experience is composed or constructed. Some experiences stick with us and, upon reflection, develop 
qualities that might not have been noticed during the situation itself. Later experiences might color the prior 
experience in a way that recasts it. Eor example, an illness may have been unpleasant at the time, but 
reflection might focus on personal struggle successfully faced, social relationships strengthened, and 
changes in outlook and habits that resulted — coloring the experience as ultimately positive and conducive 
to growth. 

• Experience unfolds over time. Experience can be seen also in the accumulation of immediate experiences 
or, moreover, as an unfolding sequence of immediate experiences that move toward an outcome. Similar to 
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the way a piece of music builds or a novel grows on you, an experience may lead to increasing complexity 
and a rewarding conclusion that depends upon the totality of unfolding events. Like immediate experience, 
this unfolding and its unfolded conclusion are also felt, and not just objects of cognition. 

• Finally, experience is historically situated. The meaning ascribed to any given experience depends in part 
on the history of previous interactions. This is a significant factor for example, as students encounter non- 
intuitive math and science principles or as they adjust to the routine of a new teacher. This notion of history 
is central to cultural-historical activity theory (CHAT) (Cole, 1996). 

Levels of Experience 

The effectiveness of an experience can be understood pragmatically in terms of the level of its potential 
impact — the degree of change it can stimulate in the near term or the growth that might continue in the future 
experiences to which it leads. This potential depends upon the quality of engagement that takes place, and can be 
demonstrated in each of the temporal dimensions described above. The levels listed below do not describe a strictly 
ordered continuum along a single variable; rather they describe common kinds of experience based on numerous 
converging conditions and qualities. 

• No experience. Given the definition of experience as transaction, not all situations qualify as experience. If 
one is unconscious of things in the world or makes no attempt to influence them, learn from them, or enjoy 
them, little or no experience occurs and no value results. 

• Mindless routine. Some experience can be characterized by the boredom that comes from forced or 
mindless routine. In this case, little investment is made by the individual, and little impact is felt from the 
situation. Growth is stymied, and the only likely impact is a developing aversion to similar experiences in 
the future. 

• Scattered/Incomplete Activity. At this level of experience, an investment of engagement is evident, but it is 
frustrated by interruptions, diversions, and roadblocks that leave it unfulfilled. Unfortunately, much of life 
can fall into this category. An individual can be quite immersed in activity, but the experience remains 
unsatisfying and unmemorable. At the end of a day filled with such experience, one might ask, “What did I 
accomplish today?” and be unable to come up with an answer other than that it was filled with activity. 

• Pleasant routine. At the level of pleasant routine, experience begins to have lasting value. Pleasant routine, 
as opposed to mindless routine, suggests significant engagement and investment in the transaction, both by 
the individual and by the situation in response. However, the growth that results from this kind of 
experience is likely to be evident only in the long term, developing incrementally and slowly. Tending a 
garden is a prime example and metaphor for this kind of experience — the routine tasks are not necessarily 
significant on their own, but an awareness of what they lead to colors them as pleasant and meaningful (cf. 
Emirbayer & Mische, 1998; James, 1976). 

• Challenging endeavors. Whether one succeeds or fails, challenging endeavors lead to significant growth 
and new knowledge about one’s place in the world. Challenge suggests substantial engagement in the 
transaction of experience, again, not just on the part of the individual attempting to meet the challenge, but 
also on the part of the world imposing the challenge. The most significant challenges come about from 
sustained effort, not instantaneous reward for confronting a difficult situation. Therefore, the nature of 
challenge is often revealed more fully in-the-making and in-reflection, even though immersion in the 
moment is also a critical characteristic. 

• Aesthetic experience. When an experience stands out from the general flow of experiences, when one can 
point to it as exhibiting heightened meaning throughout in its immediacy, its unfolding, and in reflection, 
experience reaches its highest level and qualifies as aesthetic (Dewey, 1934/1989). ‘Aesthetic,’ in this 
sense, includes recognition that works of art are refined experiences — they function as an exercise of our 
ability to derive meaning from life, and are not distinct in quality from many everyday experiences 
(Berleant, 1991). Aesthetic experience is characterized by meeting an indeterminate situation with 
anticipation and active engagement, and following through toward a unifying consummation. Aesthetic as 
peak experience can be powerful and life-changing, and at minimum intensely enjoyable and memorable. 
As the levels of experience proceed from mindless to pleasant routine through challenge to aesthetic 

experience, increased engagement is assumed. In fact, engagement may be the best indicator of the level of potential 
outcomes of an experience. The quality of engagement that develops in an experience is influenced by both 
situational and individual qualities, some of the most significant of which are discussed in the following sections. 
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Situational Qualities Influencing Experience 

The situation in which an experience takes place includes the physical world and material objects that 
afford or constrain engagement, including one’s own body. Other people are also a key part of the situation, as are 
related social and cultural qualities. The list below examines a very general set of qualities that describe these 
conditions. 

• Immediacy. Experience has an immediate temporal dimension, and therefore a key quality of situations is 
how well they absorb individuals in this dimension by offering substantive immediately felt qualities. The 
degree to which an experience becomes immediate, in the sense of un-mediated or unencumbered by 
intervening interpretation or representation, can be an important indicator of its ultimate power 
(Csikszentmihalyi, 1990). Experience, from the viewpoint of the individual, comes in waves of perceptions, 
thoughts, and feelings that arise from active engagement with a situation. To submit to these waves, even 
when they occur without apparent organization, is to appreciate the immediacy of an experience. Merely 
considering a situation in the abstract or passively observing it without immersion will have lesser impact. 
Designers achieve immediacy by attending to the emotional authenticity and sensual qualities of a situation 
or product, and when the forms and textures of experience they offer are consistent with an unfolding 
meaning. 

• Malleability. As a transaction, effective experience requires give and take, both doing and undergoing on 
the part of the individual involved. Experience and what we gain from it relies heavily upon what we bring, 
what we contribute to its unfolding, and how we think about it upon reflection. In turn, situations that 
provide for effective experience will be malleable, or open to the contributions of individuals. Situations 
conducive to powerful experience leave room for individualized engagement and ownership, even if only 
by remaining open to interpretation, as is the case with otherwise finished works of art. Malleability is also 
the quality that allows experience to be composed over time from the raw materials of immediate 
experience. To be malleable is more than being responsive, suggesting an explorability or openness to 
taking shape — giving experiences an adventure -like quality. In this sense, malleable situations are to a 
degree provisional, with their final meaning deferred and begging additional action and/or interpretation. 

• Compellingness. By presenting something of novelty or interest, a compelling situation invites entrance and 
pursuit. It draws a person in; it leads a person along or propels a person forward. Compelling situations 
invite movement and response by presenting a provocative idea, challenge, issue, or conflict in need of 
resolution. This relates to the notion of problem solving in cognition and instructional design (Jonassen, 
2000), but extends to a more general way people encounter the world (Wong, 2007). While searching for a 
better noun, we are using the term “compellingness” to describe this quality. 

• Resonance. Experiences show varying degrees of influence and persistence, but they never just stop at the 
edges of the situation. We carry experiences within us and into new situations, continuing to reflect upon 
their meaning, allowing it to develop contours and depth as it resonates with other aspects of our lives. The 
richest experiences can change the timbre of the other situations we encounter with the knowledge we’ve 
gained and new points of view we’ve adopted. If their resonance is sufficiently strong they may recast 
nearly everything we do and may impact our lives indefinitely. When situations are too scattered and 
incomplete or too closed in and too pat in what they have to tell us, they can fade quickly. Situations gain 
resonance by connecting to our current lives and by leaving a residue of ideas and attitudes that can attach 
to the future situations we touch. Resonant experiences leave us energized to ponder them further and to 
look for future connections. 

• Coherence. Much of experience is disjointed and seems to move from event to event without connection or 
meaning. More rewarding experiences feel unified and coherent — they hang together. To judge something 
as coherent is equivalent to saying that something is meaningful — either it connects to our lives and 
intentions in a way that completes previous experience, or it reveals a high degree of internal unity and can 
be appreciated on its own terms. As it is with works of art, those experiences that reveal significant 
coherence of intent in the midst of threatening chaos, or those that achieve successful unification of widely 
disparate elements, are often those judged most rewarding. Experiences richen when one has to struggle 
toward consummation (Dewey, 1934/1989). 

Instructional providers can aspire to high degrees of immediacy, malleability, compellingness, resonance, 
and coherence in the learning situations they create. Attention to the textures of experience, providing opportunities 
for learners to mold a situation, supplying enticing or forceful ideas, showing connections to other experience and 
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pointing to future experience, and creating learning activities that move in concert toward a consummation of 
growth in the learner — each of these is critical in allowing a powerful learning experience to develop. 

Qualities of Individuals Influencing Experience 

What an individual brings to a situation influences the experience as much as its situational qualities. To a 
large degree, each individual creates a unique experience with herself at the center. While instructional designers are 
often primarily concerned with influencing situational qualities and with how learner qualities mesh with these, they 
always bring personal qualities to the learning experience that also impact its effectiveness. Some of the more 
important individual qualities are described below. 

• Intent. Each person has a particular orientation to the world indicated by the goals and interests (intentions 
in common parlance) they carry into a situation. But intent goes beyond this notion of having an express 
purpose. Intentionality also suggests the inherent world-directedness of consciousness (Stewart & 
Mickunas, 1990), a fundamental relationship with the world that includes our attitudes, values, hopes, 
beliefs, likes, and dislikes, as well as assumptions about one’s place within the world (Husserl, 1982/1999). 
Clearly, the concept of intentionality covers a lot of ground that psychology typically tries to sort out 
(Anderman & Wolters, 2006). One’s intent has an important impact on the experience that develops in a 
situation, and the intentions of other individuals involved (such as instructors and IDs) will also have an 
impact. When a person exercises conscious intent, an experience has a higher chance to develop to its full 
potential. 

• Presence. One’s ability to have an experience and the resulting level of that experience depend on the 
degree of presence one brings to it. Presence begins with being-there, which includes physical and mental 
presence, leading to sufficient engagement with the content of a situation for understanding to occur. But 
presence also includes the important quality of being-with, a willingness to engage with the other 
individuals in an experience at multiple levels. Being-with includes a willing vulnerability that supports 
others — the practice of empathy rather than focusing entirely on one’s own intentions. Eor example, with 
an empathic stance, one encourages an open exchange of thoughts and feelings rather than professing a 
single point of view. Presence also includes being-one ’s-self, an attitude of authenticity and genuine 
expression of one’s own thoughts and feelings. Being-one’s-self is important for connecting with others 
and having a genuine relationship to the situation in general (Heidegger, 1962). A person lacking this 
aspect of presence may be unable to learn from an experience because he never admits to the need to learn. 
As an educational goal, enhancing the learning experience entails two related outcomes: helping learners 
(1) become more fully alive and aware in the present and (2) be more completely responsible agents who 
can draw on their pasts to project, imagine, and make things happen in the future. 

• Openness. Openness doesn’t imply passively giving in to any external force, but instead possessing the 
willingness to submit to being changed while maintaining personal integrity (Dewey, 1916). The ability to 
change in productive ways requires such integrity combined with openness. Openness is also a type of 
dependency that “denotes a power rather than a weakness” because it creates interdependency and 
increased social capacity (p. 44). Openness is an essential state for developing engagement within a 
situation. 

• Trust. Trust encompasses several essential qualities of the individual in an experience. It suggests a degree 
of faith that positive outcomes can emerge from a situation — a willingness to “suspend disbelief,” 
demonstrate patience, and extend one’s presence without immediate reward. It also describes having 
anticipation, a mental and emotional attachment to the situation that looks ahead to outcomes based on 
indeterminate conditions that need resolution. Einally, trust always contains an element of forgiveness that 
although the situation and individuals will occasionally fall short of expectations, things can be repaired 
and reconciled and differences bridged (Wilson, 1999; Song, Hannafin, & Hill, 2007). 

While these qualities relate to the individual learner, they fit all individuals participating in the situation. 
This includes other learners - but also the instructional designer, the teacher, and others involved in the learning 
transaction. 

The dimensions and qualities of experience posited here are not assumed to be entirely independent. 
Instead, they are to a degree overlapping, mutually influencing, and interdependent. However, even a non- 
orthogonal framework is useful in providing structure for instructional design judgment. Because experience is 
highly contextual, and therefore subject to cultural influences, the framework is also likely, to a degree, influenced 
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by Western values. For example, while the framework suggests the importance of situational malleability and 
individual intent in enhancing experience, in other cultures these qualities might be de-emphasized in favor of 
stability and social harmony. 

In keeping with the transactional nature of experience, one can see both parallel and complementary 
relationships between the individual qualities in an experience and the qualities of the situation. For example, 
individual intent can be seen as parallel to the resonance of the situation — they are qualities that reach outside the 
experience. The complement of intent — the situational quality that allows it to be exercised — is malleability. An 
additional parallel relationship is that between individual presence and situational immediacy (how a situation 
exudes its presence), and both individual openness and trust have a parallel in situational malleability (a trust that 
control is not required). Additional complementary relationships include those between individual presence and the 
situational malleability that can admit it, individual openness to a situational resonance, and individual trust (with its 
expectation for meaning) that situational coherence will become evident. The compellingness of a situation can 
overcome initially limited individual intent, presence, or trust by stimulating attention and interest. 

Figure 1 depicts the relationships between situational and individual qualities of an experience and the 
levels of experience. Increasing levels in each of the qualities will in turn lead to a higher level of experience. 
Experience exhibits peaks and valleys through time depending on the convergence of these qualities. The 
relationship between intent and experience has additional complexities — intentions must also match to a sufficient 
degree what the situation has to offer. 




The characteristics of situations and individuals impacting experience are of course more numerous than 
those listed above. For example, each experience also has a particular content or set of contents that are most salient. 
In complement, individuals come with a particular level of knowledge and prior experience with that content, which 
will determine the quality of learning that occurs. This relationship is one that instructional providers understand 
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quite well, and so it has not been treated here directly. However, the quality of intent does capture entry conditions 
such as goals, attitudes, and beliefs in regards to content, and malleability suggests situational accommodation to 
these and to a learner’s level of content knowledge. 

Examining Learning Experiences 

With the framework now in place, we revisit the two learning experiences described earlier, applying the 
framework to analyze their qualities. 

Pat’s Psychology Course Revisited 

The most salient qualities of this learning experience are my own limited intent, presence, and openness, 
demonstrated by my having taken the pass/fail option and refusing to mitigate the effects of the large lecture-hall 
environment by sitting closer to the instructor and asking or answering questions. While I brought a level of interest 
to the course, I did little to build upon it. The instructor herself fell victim to the large lecture medium by offering 
limited opportunities for interaction or engaging activity, and by keeping her professorial distance. The course was 
predictable, and largely trustworthy, but at the expense of engagement. 

The course situation created by the instructor offered little to increase immediacy or resonance. Pew 
interesting media or activities broke the lecture-hall impediments to engagement, and the large class size limited 
discussion that could make the course content more relevant to students’ lives. Malleability was never a goal for the 
course, or at least I did not recognize opportunities to poke and probe and engage with course elements. There 
appeared to be little I could do to contribute to the structure or even the details of the experience, other than perhaps 
to ask a question during a lecture. Being a young adult just leaving behind adolescence, the content on human 
psychological development offered significant resonance, and was at times inherently compelling in explaining the 
drama of maturation, but I don’t recall these qualities being used to advantage. Coherence was achieved only by 
“sticking to the text,” but it was a coherence with little reward, one created more through routine than through 
challenge or aesthetic experience. 

In reality, the only memorable aspect of the learning experience was created by my decision to violate 
protocol and skip the final exam. The engagement that resulted during the post-semester meeting with the professor 
and while completing the final, essay-based exam in a way salvaged what would otherwise have been a completely 
weak experience. 

Erin’ s Adventure 

This learning experience was designed in direct response to the limitations imposed by courses like 
Psychology 302, so it shows many opposite qualities. Eor example, while malleability was almost entirely absent in 
the lecture hall, the Arctic Transect program was predicated upon offering teachers and students the malleability 
necessary to customize, and even personalize, the curriculum, even though it was primarily focused on a remote 
part of the world. Erin’s teachers could choose curricular components and activities according to how well they felt 
these fit their existing curriculum and student preferences; the problem-based design of the materials called for 
active student engagement in guiding the outcomes and even many of the particulars of instruction. Erequent chat 
sessions and online collaborative zones gave Erin additional powerful opportunities for making contributions, which 
in turn increased the resonance of the experience. 

The immediacy and, ultimately, the coherence of the experience were enhanced by the linkages to the real- 
time expedition. The subject matter came alive for Erin as she viewed photos and videos and read stories from those 
on the expedition. The naturally developing drama of the dangerous trek and anticipation of its completion became 
compelling, creating emotional impact and visceral appeal. 

Even though many of the interactions in the course were had with remote participants and experts via the 
Web, the program was designed to increase the presence of all participants through structured, moderated 
discussions. Erequently emails to teachers and students from “Education Basecamp” kept the program activities and 
expedition in front of participants. The open forums and chat sessions, as well as the collaborative areas, 
demonstrated a high degree of trust that students could make constructive contributions and share personal stories 
and ideas. While Erin may not have come to the experience with a high degree of interest or intent to learn, the 
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connections she made to the content and other participants (including, of course, the sled dogs) created a developing 
intent that made her highly engaged and has continued to grow beyond the end of the experience. 

Conclusions 

The framework reveals experience as temporally grounded, multi-dimensional, and systemic in nature. This 
of course places high demands on anyone attempting to research the learning experience that results in any given 
instructional situation. Developing a clear picture of learning experience calls for a variety of research 
methodologies, including phenomenological and ethnographic techniques. Narrative approaches to research 
(Polkinghome, 1988) will be useful for capturing a rich description of the qualities a learner brings to bear in 
response to the qualities of learning situations, and how these meet in the unfolding story of the experience (as well 
as narrative approaches to design - see Dickey, 2006). Mixed methods approaches (Onwuegbuzie & Leech, 2004) 
are also appropriate because the examination of quantifiable learning outcomes is seen as critical by many 
stakeholders in educational contexts. Observation protocols borrowed from research in technology usage might 
provide an additional objective measure of experience (Axelrod & Hone, 2006). 

Our research lab (IDEAL Lab, no date) is currently exploring ways to better track and capture the 
conversational nature of instructional transactions over time. Simultaneously, instructors and students are engaging 
each other and observing and noticing impacts and changes. A research protocol that collected data from instructor 
and students over the course of a unit or term, then noting differences and converging perceptions, could help 
capture a multi-party transaction and get a window into people’s experiences — especially when combined with 
after-the-fact reflections and stories. 

In a similar spirit, Parrish and Botturi (2008) performed research to understand how learning engagement 
evolved in two distinct instructional situations in which conscious attention was applied to increasing the aesthetic 
qualities of the learning experience. Their results showed high variability among the individual and group patterns of 
engagement, and evidence of highly individualized responses to the same instruction, reinforcing the ideas that peak 
aesthetic experience is somewhat idiosyncratic and subjective, and cannot be strictly designed by arranging the 
situation in a particular way. 

Because research in this area has a dual focus of understanding processes and improving instructional 
outcomes, we suggest action research as a viable strategy. Action research engages participants with the intent of 
improving outcomes and effectiveness and effecting social change - goals very much in line with efforts to 
understand learning experience in real environments. 

Activity theory is an existing framework that can describe the transactional nature of a learning experience 
(Cole, 1996; Jonassen & Roher-Murphy, 1999; Lemke, 1997), and so it might seem to offer a sufficiently strong 
starting point for research. However, activity theory, while accounting for the individual goals and intentions of the 
actors within activity systems, sees them as somewhat disembodied. Also, activity theory does not concern itself 
with level of experience as a critical component in an activity system. The framework provided here deepens 
activity theory’s materialist, “outside looking in’’ account of learning by adding an “inside looking out’’ perspective. 

A number of other theories can assist in formulating the experience construct. Working largely from a 
socio-cognitivist perspective, educational game theorists have explored engagement and the learner experience 
(Dickey, 2005; Gee, 2003). Also as mentioned, traditional design fields have explored the user experienced 
(McCarthy & Wright, 2004). Other contributing theory bases for understanding experience include sociological 
theories of agency (Emirbayer & Mische, 1998 - see also Greeno, 2006); pragmatist philosophy of action and 
reasoning, particularly ideas of John Dewey, William James, and George Herbert Mead; practice theory (Bourdieu, 
1990; Giddens, 1984; Kemmis, in press); critical theory in general (Dimitriadis & Kamberelis, 2006).; and 
conversation theory (Pask, 1976). Most of these areas are not well represented in the instructional-design literature, 
and their inclusion could prove fruitful for the field. 

In addition to theory, examination of best practices can yield insights and provocations. Public school 
teachers have received recognition for outstanding efforts to bring classrooms alive with drama, adventure, and 
challenge. The Hobart Shakespeareans are an example (see their website), as is Albert Cullum, profiled in the PBS 
Independent Lens film, A Touch of Greatness (see website reference). 

Learning experience as a central concern for instructional providers is an innovation in several ways. Close 
attention to and valuing of learning experience leads to a crafting stance toward practice (Sennett, 2008). And 
greater attention to craft leads to additional care and responsiveness in implementation. One way of nurturing this 
attitude is to approach the instructional design task as one of composing a narrative of learning experience (Parrish, 
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2006), using story writing as a tool to imagine and better understand the complexity of the experience. Consideration 
of the temporal dimensions and situational qualities offered here suggest that multiple approaches to content 
presentation, learning activity design, and context and relationship building are critical. The framework of 
experience provided here calls for an increased accounting of the qualities learners bring to an instructional situation 
and strategies for encouraging higher levels of positive qualities. Finally, the framework might remind instructional 
providers of their own roles as individuals contributing to learning experiences, and ask them to consider their own 
intentions, levels of presence, openness, and trust. Instruction and learning become as much about relating and 
connecting as about knowledge-demonstrating. 

Aesthetic experience, the level of experience that potentially offers the deepest and most lasting impacts, is 
not an inexplicable occurrence. It is merely a powerful convergence of high levels of each of the qualities of 
experience discussed above. Aesthetic experience is worth striving for in instructional situations not only for 
developing situational motivation and engagement, but because it can spill over into other parts of learners’ lives, 
offering an empowering anticipation of the potential to be found in any new experience (Dewey, 1916; Wilson, 
Switzer, Parrish, & Balasubramanian, 2007). It can instill a desire for the rewards of attending to the immediacy, 
engaging the malleability, ensuring compellingness, and finding the resonance and coherence in experience. It can 
also inspire learners toward higher levels of presence, openness, and trust, and stimulate healthy intent. On the other 
hand, repeated experiences that don’t allow these qualities to manifest themselves may cause learners to shut down 
to the potential of experience and growth. 
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Background 

Few educational researchers study scientific or medical journals for knowledge or insight into either the 
theory or the practice of education. Educators might ask if science informs educational research. We might ask if 
modern science lends credence to existing educational theories. Or, does science provide insights into the 
formulation of new theories of education or of the mind? 

In this paper, I will review recent scientific studies of the human brain that exemplify specific stages of 
research into human brain physiology and function. I will examine the results of these selected studies to determine 
how they inform educational theory, using Piaget’s Theory of Cognitive Development as an example. Finally, I will 
consider how educational theories, as exemplified by Piaget’s, could inform and direct the scientific investigations 
of human brain physiology and mental development. 

Studies of Brain Physiology 

Although Nuclear Magnetic Resonance Spectroscopy has been an analytical tool for more than fifty years, 
it is only recently that a variant. Functional Magnetic Resonance Imaging [fMRI] has been applied to in situ studies 
of the human brain. Pans (Pans, Zijdenbos, Worsley, Collins, Blumenthal, Giedd, Rapoport, & Evans, 1999), 
hypothesizing that the structural maturation of individual brain regions and their connecting pathways is a 
physiological condition necessary for the successful development of cognitive, motor, and sensory functions, 
reasoned that the smooth flow of neural impulses throughout the brain would be integrated across the spatially 
segregated brain regions affecting these functions. Using fMRI, they studied the maturation of fiber tracts in the 
brains of 1 1 1 children and adolescents ranging in age from 4 to 17 years. They concluded that the increases in white 
matter that they observed corresponded to increased mental capacity in older children and that these increases were 
irreversible. [Figures 1 and 2] 

Sowell (Sowell, Thompson, Tessner, & Toga, 2001) used fMRI to study patterns of brain maturation 
between children and adolescents and between adolescents and adults, creating maps of the spatial distribution of 
cortical gray matter density that not only confirmed Pans’ analysis of brain growth, but also demonstrated significant 
changes in the frontal areas most responsible for higher order thinking. Their analyses showed marked increases in 
white matter in those areas and a simultaneous decrease in gray matter, relating increases in analytical and symbolic 
thinking skills to physiological changes in the brain. [Figure 3] 

Geidd (Giedd, 2004) also used fMRI to study dynamic changes in brain anatomy from childhood 
throughout adolescence, observing that gray matter increased throughout childhood, but fell rapidly after puberty, 
creating an inverted U-shape. Geidd also observed that white matter in the brain increased in the brain throughout 
childhood and rapidly increased during adolescence and continued into adulthood. 

In 2004, Nagy (Nagy, Westerberg, & Klingberg, 2004) expanded on Giedd’s observations. They 
determined that the process of brain maturation involved significant growth not only in the white matter of the 
prefrontal regions but also of the corpus callosum that links and integrates the functions of the two sides of the brain. 

Most recently, Whitford (Whitford, Rennie, Grieve, Clark, Gordon, & Williams, 2007) used fMRI 
combined with electroencephalography recordings to study subjects ranging in age from 10 to 30 years. They 
reported that gray matter volume decreased across the age bracket in the frontal and parietal cortices, with the 
greatest change occurring during adolescence. Concomitantly, the observed that white matter volume in the parietal, 
frontal, temporal and occipital lobes increased linearly with age. 

Studies of Brain Function 

As the above researchers were studying the changes in brain morphology, other researchers were 
investigating the effects of these changes on thought processes and behavior. In 2005, Barnea-Goraly (Barnea- 
Goraly, Menon, Eckert, Tamm, Bammer, Karchemskiy, Dant, & Reiss, 2005) investigated white matter maturation 
as reflected by changes in anisotropy and white matter density with age. They concluded that white matter 
maturation especially of the prefrontal cortex is related to the development of specific cognitive functions as well as 
sensorimotor processes. 
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In 2006, Rubia (Rubia, Smith, Woolley, Nosarti, Heyman, Taylor, & Brammer, 2006) used rapid 
randomized mixed-trial event-related fMRI to investigate developmental difference of the neural networks 
mediating a range of motor and cognitive inhibition functions in adolescents and adults. They reported, “Whole 
brain regression analyses with age across all subjects showed progressive age-related changes in similar and 
extended clusters of task-specific frontostriatal, frontotemporal and frontoparietal networks. These findings suggest 
progressive maturation of task-specific frontostriatal and frontocortical networks for cognitive control functions in 
the transition from childhood to mid-adulthood.” 

Conclusions Based on Scientific Studies 

The experimental results reported in these studies substantiate that age-related processes are responsible for 
the maturation of the human brain. In the early stages of life, the child’s brain, composed primarily of gray matter, 
grows in size, reaching a maximum at approximately age twelve. Throughout this period, white matter gradually 
replaces gray matter in a specified sequence, moving from the base of the brain into the mid-brain. 

Puberty appears to initiate complex structural changes in the adolescent brain. Gray matter volume 
decreases in the process called synaptic pruning. The remaining gray matter develops into white matter through the 
process of myelination, increasing transmission rates up to 100-fold. This process continues into early adulthood and 
perhaps beyond. 

Piaget’s Theory of Cognitive Development 

As educators, we are familiar with Piaget’s experimental approach to understanding the mental 
development of children and adolescents (Inhelder & Piaget, 1958). Let us recall one experimental sequence in 
particular, namely the two-pan balance. [Figure 5] Young children, ages 4-7, were unable to balance the two pans. 
Prepubescent children could only succeed by placing the pans on the extreme ends of the balance beam. [Figure 6] 
Immediately post pubescent adolescents were able to balance the beams using different combinations of pan 
positions and weights, but were unable to explain why. Adolescents of ages 14 to 16 years were able to balance the 
pans and were able to explain the relationships between relative distances and weights in the pans. [Figure 7] 

Piaget developed three hypotheses from these and similar studies that became the foundations for his 
Theory of Cognitive Development. He suggested that children passed through a specified developmental sequence 
in the same order, and that this sequence was irreversible. Finally, he hypothesized that this developmental sequence 
was the result of physiological changes in the human brain. 

Science Informs Educational Research 

The studies reported provide support for Piaget’s hypotheses. Every study reports that the human brain 
undergoes a predictable and unidirectional series of changes throughout childhood and extending through 
adolescence and into adulthood. These changes involve increasing brain size in children; increased density of gray 
matter in children; synaptic pruning and myelination in adolescents and young adults. These reports also 
demonstrate that these changes are irreversible, especially those of synaptic pruning and of myelination. [Eigure 4] 
Educational Theory Informs Science 

Educational research is replete with experimental evidence of learning processes, implying brain functions. 
However, until recently, there has been no way to directly observe the functioning human brain. Now that medical 
science has tools such as fMRI, it would be important to attempt to reproduce educational research to observe the 
not only the behavior but also the brain’s operations while performing various experiential and learning tasks. 

Eor instance, it would be possible to reproduce Piaget’s experiments with children while they were being 
observed by fMRI. Obviously, there are difficulties to be overcome do to the physically large and constricting MRI 
apparatus and its large magnetic field. However, educational technologists have developed computer-aided learning 
systems and simulations that should be useable even under such severe space and electromagnetic restrictions. 

Such a combination of educational and scientific research could begin to answer vexing questions in 
learning, education and brain physiology. Could we explore the thinking process, learning what areas of the brain 
are used or changed? Could we correlate the growth of specific areas of the brain with increasing ability to learn or 
to solve problems? Could we observe the effects of synaptic pruning on learning and memory? Could we observe 
the effects of myelination on mental development and maturation? Could be observe how memory is created, 
developed and accessed? 

Conclusion 

Science has informed education and educational theory, providing new avenues of exploration for 
educators. Educational theory can inform and direct science to a greater understanding of human learning processes. 
The close cooperation of educators and scientists would result in a greater understanding of the human brain, leading 
to more efficient and more effective educational processes. A more highly educated populace would promote a 
greater understanding of ourselves and of our universe. 
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FIGURES 



Figure 1. White matter density in left and right internal capsules and arcuate fasciculi by age. 

(Paus, Zijdenbos, Worsley, Collins, Blumenthal, Giedd, Rapoport, & Evans, 1999, p. 3) Used by Permission. 
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Figure 2. Age-related changes in white matter density along the putative corticospinal tract 
The thresholded maps of f-statistic values [f> 3.0] are superimposed on the axial [A] and coronal [B] sections 
through the MR image of a single subject. The location of the changes in the posterior limb of the internal capsule 
suggested that the changes involved the corticospinal and, possibly, thalamocortical tracts. 

(Paus, Zijdenbos, Worsley, Collins, Blumenthal, Giedd, Rapoport, & Evans, 1999, p. 3) Used by Permission. 
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3 5 7 9£ Figure 3. Composite 

statistical maps of age effects 

A. Relating brain growth 

[Green] with gray matter loss [Blue] in the child-to-adolescent group. Areas in red show a combination of brain 
growth and gray matter loss. Note; Growth in anterior brain and gray matter loss in all four lobes of the brain. 

B. Relating brain growth [Green] with gray matter loss [Blue] in the adolescent-to-adult group. Areas in red show a 
combination of brain growth and gray matter loss. Note: Growth in frontal and medial lobes and gray matter loss in 
the frontal lobes. 

C. Left, top and right brain, highlighting differences in correlations between brain growth and gray matter loss in the 
child-to-adolescent group. The areas in red are those of greatest growth while also those of greatest reduction in gray 
matter density. The white areas are those of growth and of increasing gray matter density. Note: Gray matter loss in 
occipital and parietal lobes. 

D. Left, top and right brain for the adolescent-to-adult group. The areas in red are those of greatest growth while also 
those of greatest reduction in gray matter density. The white areas are those of growth and of increasing gray matter 
density. Note: Gray matter loss in frontal lobes, associated with higher thought functions. 

(Sowell, Thompson, Tessner, & Toga, 2001, p. 8825) Used by Permission. 
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Adolescent to Adult Gray vs. DFC 
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Brain Mass 



Figure 4. Chronological Age and Piagetian Stage vs Brain Mass 
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Figure 5. Simple Two-Pan Balance 




349 



Figure 6. Two-Pan Balance as Balance by Pre-Pubescent Child 
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Figure 7. Two-Pan Balance as Balanced by Adolescent. 
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Abstract 

The implementation of Virtual learning environments depends on decisions related to the educational design and 
on other decisions related to technology itself and the selection of the most appropriate system and tools. This 
selection requires thinking about instructional design principles, knowledge of the existing technology and their 
educational possibilities. 

Sometimes when an institution plans to implement a virtual learning environment, responsible does not have 
knowledge of both the pedagogical and technical areas. Usually the selection of LMS is based principally upon 
technical criteria. 

Because we believe that technology must be used to facilitate the teaching/learning process, we designed a tool 
made for decision making about the technology required for implementing learning strategies, keeping in mind 
three points of view: pedagogical, technical and organizational. For this, we tried to determine the indicators and 
criteria to have in mind by users and organizations selecting the most appropriate technologies for the high 
quality e-learning experience. 



Keywords 

Learning Management System selection, virtual learning environment, e-learning 

1 . Introduction 

Nowadays through Internet, stakeholders and teachers find many tools at their disposition for the teaching - 
learning process. With the proliferation of these tools, the problem is more in determining which tools are the 
most appropriate for obtaining specific educative goals. Following the same line of thinking as McGreal, Gram 
& Marks (1998), Spector, Wasson & Davidson (1999), Cook (2000), we believe that taking into account the 
instructive design principles based on networks and the knowledge of both the technical characteristics of the 
tools and the pedagogical aspects, would contribute to selecting the most appropriate software for the 
distribution of educative experiences based on networks. 

However, selecting a Learning Management System (LMS) is very complex due to the great number of tools 
developed specifically for distribution and management of virtual environments aimed at learning, and also due 
to the continous advances made and the upgrading of new tools which this field is generating. 

The dynamic use and the increase of the ITC in learning, the expansion which is now happening with the results 
of a multitude of experiments, the emerging models of teaching, the immense quantity of tools and applications 
developed, the lack of knowledge about the educative possibilities of these, the disregard of functionalities and 
technical possibilities and the lack of criteria for evaluating and selecting tools, has led us to this research. 

2.- Description of the Research Process 

This study was conducted as a design research (van den Akker, 1999; Reigeluth & Frick, 1999; Reeves, 2000), 
and by identifying the indicators and criteria which need to be taken into account when selecting tools, we have 
developed an instrument which helps to make decisions about the most suitable technical support needed. Using 
the Development and Research methodological approach we have built a tool to help in the making decisions 
process for selecting technological framework to implement virtual learning environment. 

Because the choice of tools cannot be made only by using one criterion, being there so many elements to take 
into account in this environment, we have based ourselves on the multi-criteria theory as our base method for 
designing the instrument. 
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We designed the instrument in two phases: 

1.- The first phase consisted in the identification of the criteria quality which defines virtual teaching- learning 
environments and also the determination of the indicators and attributes which needed to be taken into account 
for the choice of platforms or tools, described above. We made this study by the functional analysis of both the 
technical possibilities of the tools and the relation to learning contexts. In relation with the functional analysis 
about technical possibilities, we made a meta-analysis (de Benito, 2006) of comparatives studies or reports 
about platforms. This allowed us to know the technical and pedagogical possibilities of the tools, and also a 
questionnaire was sent to professors with e-learning experience. For the functional analysis in relation to the 
learning context we used a part of the questionnaire mentioned above, aimed to analyze the different learning 
contexts in virtual environments, (de Benito & Salinas, 2005; de Benito & Salinas, 2006, Salinas & de Benito, 
2007) 

2.- The second phase was the design, development and validation of the prototype. Once validated by experts 
the instrument was made up of 20 questions relative to organizational, pedagogical, technological and 
economical aspects of a virtual formation environment (de Benito & Salinas, 2007). 

3.- Identification of Indicators and Criteria for Selection 

With the purpose of identifying the indicators and criteria which need to be taken into account when selecting 
tools, we made a functional analysis of the technology in relation to the learning contexts. 

This functional analysis includes: 

Analysis of didactic techniques used by the teachers 

Analysis of didactic techniques from the point of view from which teaching model was 
implemented 

Analysis of didactic techniques in relation to the type of studies 
Analysis of didactic uses of technology 
Matrix of didactic functions of the tools 

Functional map of tools, from a technological point of view through a meta-analysis of 
comparatives studies or reports about platforms and also a questionnaire was sent to professors 
with e-learning experience. 

Some of the results obtained from the functional analysis revealed what didactic functions the tools presented. 
Studying the different strategies implemented, allowed us to find out which tools were used more often in 
virtual environments, how they were used and also to make a matrix of the didactic functions of these tools. 

3.1 Analysis of Didactic Techniques 

The didactic techniques in use by CMC have been widely analysed by various authors, the number of techniques 
in use is large and even more if we consider the variables which can be incorporated. So in order to be able to 
analyse the different didactic functions of these tools we decided to select and group together those techniques 
which we found were used by the greatest number of teachers, giving the person questioned the option of 
incorporating others which we had not considered. 

As reflected in graphic 1, the most often used didactic techniques were: availability of material and learning 
resources for the student (85%), followed by tutorials (79%) and the search and recovery of information (73%). 
The least often used techniques were those based on simulation, games and roll playing. 
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1 access to material 

2 search and recovery 
of information 

3 online presentations, 
conferences 

4 consulting experts 

5 demonstration 

6 debates and 
discussion groups 

7 simulation, games 
and roll playing 

8 informal exchanges 

9 group activities 

10 case studies, problem 
solving 

1 1 working on projects/ 
web quest 

1 2 individual tutoring 

13 group tutoring 



Graphic 1 : Percentages didactic techniques used. 



We also analysed the didactic techniques from the point of view from which teaching model was implemented. 
In this case we observed that the techniques are almost the same independently of the teaching model though 
distance model also gave importance to group tutoring. 



Face-to-face 


Blendend 


At distance 


- Access to material 82% 

- Search and recovery of 
information 74% 

- Individual tutoring 65% 


- Individual tutoring 89% 

- Access to material 78% 

- Debates and discussion 
groups/symposium, round 
table or panel 72% 


- Access to material and 
individual tutoring (100%) 

- Search and recovery of 
information, debates and 
discussion groups, 
symposium, round table or 
panel (83%) 

- Group tutoring (78%) 



Table 1. Resume of the most used didactic techniques in function of the teaching models 



Analysing the techniques in relation to the type of studies in which they are implemented (Health Sciences, 
Experimental Sciences, Social and Law Sciences, Teaching and Humanities) we observed that access to material 
and the search and recovery of information were the techniques most used independently of the subject matter 
studied. Those who use the greatest number and variety of techniques are those who study experimental sciences 
followed by those who study social and law sciences. 

Online presentations, conferences or consulting experts were used by approximately 50% of professors of 
experimental sciences, teaching techniques and humanities, whereas consulting experts is hardly used at all by 
those in the Health science field. 

Demonstration techniques were used very little in general (37% of all the professors who answered the 
questionnaire) though it is to be noticed that they were the least used in technical teaching. 

So though debates, discussion groups etc. were the most used techniques, only 60% implemented them 
sometimes. As to the subjects in which they were most used, experimental sciences came first (80%) followed 
by technical studies (78%), health sciences (66%) and last social sciences and humanities (60 and 50% 
respectively). 

Simulations were the least used generally though 60% of the experimental sciences professors say they used 
them. 

Informal exchanges cannot be called a proper didactic technique, in any case it needs to be incorporated with 
other techniques because of the fact that 50% of professors say they use it. 

Group activities, such as case studies, problem solving, etc., are used by more that 50% of the professors in 
social and law studies and experimental sciences. 
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Work on projects is a technique which has more followers every day, however it is still one of the least used 
ones say the professors who filled out the questionnaire. The subjects which most use it are social and law 
sciences, followed by health sciences. This is the technique which experimental sciences professors say they use 
least. 

3.2. Didactic Uses of Technology 

The tool’s use is another aspect of the analysis of didactic techniques. In order to analyse these, we divided the 
tools into three groups : communication, (mail, chat, forum, instant messaging, news, video conferences), 
organisational (job assignment, calendar, student groups, authoring tools, brainstorming, voting) and those 
related to access of information (shared applications and files, links to URL, FAQ, conceptual maps, shared 
navigating, electronic blackboard). 

Keeping the above in mind, we made graphics 2, 3 and 4 which show the tools used by professors in each 
didactic technique by and the percentage of which say they use the techniques specifically. We observed that the 
tools which facilitated communication were the most generally used. 

Graphic 2 show that mail and forums are way ahead of the others and are used in almost all didactic techniques. 
Mail, as is logical, is used for individual tutoring, access to information and case studies. The technique in 
which they are least used is for online presentations, conferences, debates and discussions in which the forums 
tool is most used. Another of the techniques where a considerable percentage of professors use forums is in the 
creation of social spaces for informal exchanges, and also, though to a lesser degree, in proposing group 
activities, simulations, games or roll playing, also in accessing information, case studies and problem solving. 
The next most used tool is the chat. This is most used in consulting experts and as a social space. Video 
conferencing is mostly used for online presentations and conferences but also in demonstrations and for 
consulting experts. News is also a little used tool, it is mostly used in accessing material (30% of professors), 
then for the search and recovery of information, online presentations, consulting experts and demonstrations 
(used by 15 to 20% of professors). In last place, instant messaging is the least used tool and the one less present 
in platforms. Those who say they use it, is with the working on projects technique, also in individual tutoring 
and informal exchanges. 




Graphic 2: Percentage of professors who use the various communications tools didactic techniques 

As to the tools which we grouped in the organisational category, the most used are, task assigning, calendars and 
student groups. Job assignment is mainly used in the techniques to access material and in work projects, 
followed by search and recovery of information and case studies. The Calendar is used in the majority of 
techniques, especially in accessing material. Student groups are mainly for group activity proposals, work 
projects, simulations and roll playing. It is least used, leaving aside individual tutoring, in consulting experts. 

The authoring tool is used for demonstrations, access to material and search and recovery of information. One 
should notice that it is not used in the work projects technique or group activities. Brainstorming and voting are 
hardly used at all. 
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Graphic 3 : Percentage of professors who use the various organisational tools 



Last we analysed the tools which facilitate access to and interchange of information. Of the 3 groups of tools, 
this is the group which is generally the least used. Those who use them most, with some differences, are links to 
Internet, followed by shared files. Links are most used for access to material and demonstrations. Secondly, by 
50% of professors for search and recovery of information and work projects. The technique in which it is least 
used, leaving aside tutoring, is for the social space. Shared files are used in work projects, access to material and 
proposals of group activities, least for consulting experts. Shared applications are the next most used tool, 
mainly for access to material and simulations. Conceptual maps are used only for access to material. Lastly 
FAQS, shared navigation and the electronic blackboard are hardly used at all with the exception of the use of the 
shared blackboard for online presentations, conferences and the shared navigation which has been used by a 
greater number of professors for simulations. 

100% 

90 % 

80 % 

70 % 

60 % 

50 % 

40 % 

30 % 

20% 

10% 

0% 



■ shared applications ■ shared files □ links 

□ FAQs ■ conceptual maps □ shared navigating 

□ electronic blackboard 

Graphic 4: Percentage professors who use the various tools which facilitate access to and interchange of 

Information in didactic techniques. 
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3.3. Matrix of Didactic Functions of the Tools 



From the results of the study we were able to make a framework for didactic use of the tools in function of how 
often they were used in the various didactic techniques. 



— not used 
* used by < 15% 

** used by 15-25% 
*** used by 25-50% 
**** used by 50-75% 



1 


access to material 


2 


search and recovery of information 


3 


online presentations, conferences 


4 


consulting experts 


5 


demonstration 


6 


debates and discussion groups 


7 


simulation, games and roll playing 


8 


a place for informal exchanges 


9 


group activities 


10 


case studies, problem solving 


11 


working on projects/ web quest 


12 


individual tutoring 


13 


group tutoring 
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Table 2. Matrix of Didactic Functions of the Tools 



4.- Design, Development and Validation of the Instrument 

The second phase of the research was the design, development and validation of the prototype. Once validated 
by experts the instrument was made up of 20 questions relative to organizational, pedagogical, technological and 
economical aspects of a virtual formation environment. 

Table 3 shows the differents criteria used and graphic 5 the relation existing between these criteria and 
attributes. . 
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TECHNICAL AND HUMAN RESOURCES 


economical 


COST (COMMERCIAL OR FREE PLATFORM) 


economical 


HOSTING 


economical 


TYPE OF INSTITUTION (HIGHER EDUCATION, NETWORK 
UNIVERSITIES, VIRTUAL CAMPUSES, ETC.) 


organizational 


EXPERIENCE OF THE INSTITUTION IN E-LEARNING 


organizational 


USE OF PLATFORMS 


organizational 


INSTITUTIONAL PROJECT 


organizational 


KNOWLEDGE OF THE TEACHERS IN THE USE OF 
VIRTUAL ENVIRONMENTS. 


organizational 


SUPPORT FOR TEACHER AND STUDENTS NEEDED 


organizational 


GEOGRAPHICAL AND TIME 


organizational 


NUMBER OF STUDENTS 


organizational 


STUDENT MANAGEMENT 


organizational/ 

technological 


EDUCATIONAL METHODOLOGY (DISTANCE, 

BLENDED,...) 


pedagogical 


METHODS AND STRATEGIES 


pedagogical 


PEDAGOGICAL RESOURCES 


pedagogical 


EVALUATION PROCESS 


pedagogical 


REUSE OF PEDAGOGICAL RESOURCES 


technological 


IMPORT/EXPORT CAPABILITIES 


technological 



Table 3. Criteria used in the instrument 




Graphic 5. Relationship between key criteria 



The answer to every question is associated to a particular attribute. And once all the questions have been 
answered, the tool generates a report of the recommended criteria the user needs to take into account when 
wanting to adopt a technological system. It is meant to be in an exportable file format so that the user can 
modify it and add his preferences. 



5.- Conclusion 

In this study we used the methodology of design and development with the objective of determining the 
organizational, pedagogical and technological developments that should be considered in the implementation of 
a virtual learning environment. An instrument has been constructed that helps in the decision making on the 
tools or technological configuration to implement virtual learning environments of quality. Since the selection of 
tools cannot be realised from a unique criterion because the elements to consider are many, we have been based 
on the decision making multicriterion as methodologic base on the design of the instrument. 
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This supposed, on the one hand, to delimit which are the quality criteria that define the virtual learning 
environments and on the other to determine the indicators and attributes to consider in the selection of platforms 
or tools. 

Through the implementation of the questionnaire and the functional analysis of the tools we have obtained 
information that, although we are aware that generalising these results requires caution, provides some first 
indications of how the LMS are being introduced in higher education. 

It is necessary to continue doing research to find answers to the design and the implementation of innovative 
methodological strategies in accordance with the possibilities which ICT offer us. The knowledge of 
technological advances concerning the possibilities presented by technology as to the distribution of content, 
access to information, interactions teachers- students and student-students, management of the learning 
environment etc, become essential elements of study. 
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Abstract 

Portable media devices like iPods, iRivers, and others, are becoming quite common in today’s society. Frequently, 
we see individuals from all walks of life tuning in - or out, as the case may be - to these small digital devices. 
Concurrent with the advent of portable hardware is the development of downloadable media. Two of these forms 
of downloadable programming are podcasting and vodcasting. While these media are rapidly becoming a part of 
everyday life , written laws that govern technology and it’s uses are not remaining current with the rapidly changing 
nuances of ownership and usage in the digital world. In this paper, we address the various types of pod/vodcasting 
and the legal and ethical implications of creating and using these media. 

Introduction 



For several years now, portable media devices like iPods, iRivers, and others, can be seen attached to everyone from 
teens on their way to school to business people on their lunch hour. Portable media have become so common in 
fact that the sight of a thin white cord snaking from a commuter’s ear into the pocket of his/her coat is almost de 
rigour. So what are people listening to or watching on these tiny devices? Most often, it’s downloaded music, 
movies, or television programs. Interestingly, with the advent of portable media devices, other forms of 
downloadable media have also emerged. Two of these forms are podcasting and vodcasting. 

Podcasts and vodcasts are forms of media found on peer-to-peer file sharing, where users exchange files over the 
Internet by either uploading or downloading files from individual terminals. According to Schnackenberg, Vega 
and Warner (2008): 

Podcasting is a distribution of audio files such as radio programs or music videos, over the Internet using either RSS 
or http://en.wikipedia.org/wiki/Atom (standard) drsit g 1 033 Atom syndication for listening on mobile devices and 
personal computers. A podcast is a web feed of audio or video files placed on the Internet for anyone to subscribe 
to and download. Podcaster's websites also may offer direct download of their files, but the subscription feed of 
automatically delivered new content is what distinguishes a podcast from a simple download or real-time streaming . 
Usually, the podcast features one type of "show" with new episodes either sporadically or at planned intervals such 
as daily, weekly, etc. There are also podcast networks that feature multiple shows on the same feed. The essence 
of podcasting is to create audio content for an audience who wants to listen whenever and wherever they desire. 
The term Podcast is a combination of the words “iPod” and “broadcast.” Although the word podcast is associated 
to the iPod brand of portable media players, podcasts themselves are not limited to specific brand of mobile devices. 

A derivative of the term (and idea) of podcast, is “vodcast,” also commonly referred to as a video podcast. A 
vodcast functions in much the same way as a podcast, except that instead of users downloading simply audio files, 
they download video files to their portable media players. Vodcasts, like podcasts, are also generally acquired via 
subscription and at this time, commonly for little or no cost. What makes the concepts of pod- and vodcasting 
novel is that they are not used to promote artists or for-profit works. Currently, podcasts and vodcasts feature 
lesser-known individuals and their work at no cost, simply to promote ideas and talent to which the public might 
otherwise not be exposed. 
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Currently, some pod/vodcasts are free to download and enjoy, while others are available for a fee. Musical 
podcasts have an intriguing mix of for-profit and free options. One would think that free downloads are generally 
created by unknown artists. While at one time that may have been true, now that is no longer the case. As 
formerly unknown artists are being discovered due to the vast number of hits and downloads for their free podcasts, 
famous artists are also following the trend and having some of their selections available for free, in hopes that the 
public will enjoy those selections enough to purchase more. Both independent producers and unsigned artists, as 
well as successful icons in the media industry are using pod/vodcast technology to reach the masses. 

Similarly, podcasts can also be talk shows, sports commentary, news, or other non-musical programs. Like musical 
podcasts, some of these are free and others cost the user per download. The advantage of downloading these types 
of programs, whether they are free or otherwise, is that the user can listen to what interests him/her when they have 
time for it, rather than when the media pushes the information to them. While it would seem that that free podcasts 
would be more enticing than ones that cost, for the convenience of having information available when a user is able 
to listen to it, a fee may not be all that unattractive. 

Presently, some of the audio that is available as a podcast, is also available as a vodcast with corresponding video 
files. In the not-too-distant past, podcasts were much more commonly downloaded because affordable portable 
media devices couldn’t handle video files. Now, portable media with the ability to play video is significantly more 
affordable to the public, and therefore infinitely more common. Vodcasts follow much the same genres as podcasts 
in terms of types and fee/free options, with the additional option of movies and television programs being more 
popular than music for downloading. (Of course, music videos are available for download as well.) As with 
podcasts, famous directors, producers, and actors are now not only offering some clips, shows, or films for free, but 
they are creating special programming that is available only as a vodcast. This then creates an entirely unique 
product, audience, and market, driven entirely by the development of the portable media player and it’s associated 
pod/vodcast needs. 

Of course, the entertainment, news, sports, and business worlds are not the only industries making use of 
pod/vodcast technology. The field of education is following this trend to target students, instructors, parents and 
the communities they serve. Schnackenberg et al. (2008) note that “there are now downloadable podcasts to help 
students learn how to properly pronounce words in a different language or learn English as a second language. 
Several historical websites also offer vodcasts where veterans are being interviewed and offering personal accounts 
of the wars and armed forces in which they served.” and “An instructor can opt to have his/her lectures available for 
downloading to a digital media player to be listened to or watched for review or to make up a missed class session. 
Science experiments and field trips can inspire discussions that could make great podcasts. These static, singularly 
transmitted files can then generate further discussions outside of the classroom.” Not only are these technologies 
available for use in the classroom, but software and hardware used in pod/vodcast creation are becoming accessible 
to students. Pupils in schools and universities are now able to create their own “casts” as a way to enhance the 
teaching and learning process. 

Given the pervasive use of pod/vodcast technology for entertainment, education, news, and a variety of other 
applications, it’s presence in the research literature is fairly new but surprisingly abundant. While no longitudinal 
studies are available, the results of investigations and reports found the uses and applications for pod/vodcasting to 
be varied and to have great potential. 



Podcasts/Vodcasts and the Literature 



Duke University made headlines and history in August 2004 when it handed out 20GB Apple® iPod devices to over 
1,600 incoming students. The Center for Instructional Technology (CIT) at Duke evaluated the educational benefits 
of the devices basing their findings on student and faculty feedback. The findings showed that podcasts created 
benefits for students such as flexible access to media, better support for individual learners and promoted student 
interest in the classroom. There was also less reliance on physical materials for faculty as well as students (Belanger, 
2005). Not all schools freely give out tools for creating and listening to podcasts, but as Zeynel Cebeci and Mehmet 
Tekdal point out in their study of audio learning, many students already own mp3 players capable of playing 
downloaded podcasts (Cebeci & Tekdal, 2006). Campbell echoes his in his study. He supports the use of podcasting 
at the university level saying that students will integrate scholastic podcasts into their lives easily because it is a 
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medium they are already using for pleasure. He writes “it is only natural that school stuff would mingle with other 
aspects of [the student’s] daily life (Campbell, 2005).” The two Cukurova University professors also identify some 
pedagogical benefits of using podcasts for teaching and learning. They state that listening has been the primary 
learning method for thousands of years and podcasting may be a positive alternative for students that dislike reading. 
Cebeci and Tekdal suggest that podcasting does not need to be solely for lectures. They propose that podcast 
substance could be varied and clips of songs or speeches could be inserted in with the necessary course content 
(Cebeci & Tekdal, 2006). Matthews adds that students with disabilities such as visual impairment can greatly benefit 
from recorded material as they can listen on their own time as much as necessary (Matthews, 2006). 

Boulos, Maramba and Wheeler found similar benefits when they examined podcasting as a tool for students of 
medicine. They recommended studies to examine how to integrate podcasting into e-Learning (Boulos et al, 2005). 
Saby Tavales and Sotiris Skevoulis of Pace University undertook a study that sought to find a method for seamless 
integration. They developed scenarios where a student or an instructor missed a class period. In both cases, students 
could download the day’s lecture in podcast form. Tavales and Skevoulis believe that this solution to the problem of 
unplanned absences is applicable at any level of education (Tavales & Skevoulis, 2006). 

Although many studies show benefits to teachers and learners, podcasting in the classroom does have opposition. 
Brock Read interviewed an English professor from Georgia College & State University who cautioned the use of 
iPods and podcasting without strict directions. He warns that without clear usage guidelines, iPods can become toys. 
Students need to be directed if they are allowed to use podcasts as an educational tool (Read, 2005). Another con of 
podcasting is the costs involved. Equipment needs include computers with certain requirements for hardware and 
software and a player such as an iPod. Devices to play podcasts on can be expensive and all universities do not have 
the means to provide them to students. Requiring students to provide their own devices could exclude some students 
(Matthews, 2006). 

Podcasting also brings up legal concerns. Donnelly and Berge bring up the issue of podcast ownership. If an 
instructor’s podcast follows a textbook, does the textbook publisher own rights to the podcast? If the podcasts are 
made available to only students in the classroom, such as with a course management system, the potential of 
podcasting is limited regardless of whether the content is accessed at school or at home. There is also confusion 
about whether music included in a podcast requires only a composition license or if a performance rights license is 
needed since podcasts are download and not streamed (Donnelly & Berge, 2006). 

Many benefits are associated with the use of podcasting as a tool for teaching and learning in the classroom. 
However, there are cautions for potential podcasters as well. The academic literature suggests that podcasting, when 
used correctly, can be an effective tool useful to both students and instructors. It is clear that more research must be 
done to answer some of the lingering questions regarding podcasting use for education and the laws surrounding it. 

Vodcasting shares many similarities with podcasting. Likewise, many of the benefits span both mediums. 
Vodcasting, however, has the added video component. This allows for a higher success rate in the teaching and 
learning environments because video, animation and interactive media have been shown to increase attention, 
motivation and interest (Chan & Lee, 2005). Hiirst and Waizenegger mention vodcasting when they present a 
variety of approaches for “lecture casting.” In their study, they suggest a type of lecture cast where the recorded 
audio lecture is accompanied by graphics or video. The recorded lecture becomes a type of vodcast. Their research 
proved this medium to be very effective for lecture casting (Hiirst & Waizenegger, 2006). 

Linda Herkenhoff researched vodcasting as a supplementary tool for training in the business world. She found that 
as an instrument for training and learning, vodcasting allowed trainees to learn at their own pace in whatever 
environment they chose. Multitasking became a possibility and students wishing to examine more advanced 
materials had the means to do so. She also found that vodcasting was a good fit for those who had English as a 
second language (Herkenhoff, 2006). This is similar to a finding by Robert Godwin-Jones in his study that examined 
“disruptive technologies for language learners.” Godwin Jones found that language instructors and learners could 
benefit from disruptive technologies, such as podcasting and vodcasting, because they provide supplementary 
opportunities for oral communication (Godwin-Jones, 2005). 

Vodcasting has not escaped criticism for its shortcomings. One of the most common issues is that students are not 
being trained in how to properly use the devices necessary for creating and playing a vodcast. Instructors are finding 
inadequate time to teach both the course content and a tutorial on how to use the vodcast technology (Chinnery, 
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2006). In order to make vodcasting a staple of a classroom, a standardized recording process would have to be 
developed to ensure a certain level of quality and limit access to copyrighted materials that are meant for classroom 
use only. The legal issues at hand are almost identical to those surrounding podcasting. It is unclear who owns the 
podcast or vodcast if there is copyrighted information contained in it (Meng, 2005). 

Vodcasts remain notoriously more difficult to create than podcasts. They contain downloadable files as well as 
streaming sources. Also, there are so many different types of video, varying in format, resolution etcetera, that users 
may have trouble with playability of a vodcast. Another issue of difficulty is found in the playback. While the video 
content of a vodcast can be split into sections or chapters, current handheld devices are unable to take advantage of 
this navigation feature (Ketterl et ak, 2006). Quality is another matter of much discussion. Although devices are 
increasingly portable, this shrinking of players is resulting in small speakers and screens as the industry standard. 
The outcome of this is poor audio-visual quality as well as the limited power of the devices (Chinnery, 2006). 

Vodcasting remains a growing technology with many potential uses for teaching and learning in the classroom 
setting as well as in other situations such as employee training. There are controversial issues surrounding 
vodcasting including legal questions and concerns about quality. As the technology grows, its implications will 
become more evident and the appropriate uses of vodcasting will become clear as research continues. 



Legal Issues and Ethical Dilemmas with Podcasting and Vodcasting 

Generally at no cost and open to the public, most podcasts and vodcasts are copyright free and embody Creative 
Commons (Creative Commons, 2008) principles where information can be used, modified, and shared as long as it 
is not for personal gain or profit and proper credit is given to the person(s) who created the material. All 
information in the public domain can be freely used to create pod/vodcasts (i.e. Aesops fables, the laws of gravity, 
Shakespearean plays, a variety classical musical scores, etc.). However, in the case of performed works, the scores, 
scripts, etc. are freely used, but the various performances may not be. Similarly, well-known television and radio 
programs (i.e., CNN, NPR) also have free podcasts and/or vodcasts available to their audience in order to 
supplement or extend their shows. (These are not to be confused with real-time streaming video, which CNN, 

NPR, and others also provide.) As well, over the last few years, many professors and corporate organizations have 
made lectures and other informational materials available for downloading to their constituents. Often, this 
information can only be accessed with a password, but it is generally still at no extra cost to the user. While each of 
these examples illustrates proper and legal use of podcasts and vodcasts, misuse also exists. 

One of the most high profile cases of the misuse of vodcasting in particular is with the Japanese anime industry. 
Recently, fans were uploading and posting to You Tube (www.youtube.com) a wide variety and styles of anime. 
Given that the anime industry in Japan, like most countries, is a for-profit business worth large sums of money, these 
illegal uploads (and subsequent illegal downloads) were impacting profit margins. Not only that, but in Japan the 
animation industry is highly regulated by government guidelines and laws because it is a very important and deeply 
embedded part of the culture. From the perspective of the Japanese government, the illegal uploading and 
downloading of anime involves more than just money, it is a matter of honor. While this specific situation was 
ultimately resolved to the satisfaction of the Japanese government after they formally requested that the illegally 
uploaded anime was found and removed from You Tube (most, but not all of the anime is now gone), the situation 
poses an intriguing legal and ethical dilemma. While by its very design, the technologies of podcasting and 
vodcasting were meant to be used freely and legally, they often are not, thereby impacting someone or some entity’s 
opportunity for recognition and money. In addition, and perhaps more interestingly, these illegal uploads and 
downloads may also inadvertently trample the mores or decorum of a location or culture with which we as users 
think we have no interaction or influence. While clearly this is not the case, it certainly illustrates the need for 
caution with podcasting and vodcasting from not only a legal standpoint, but also a cultural one. 

While some of the misuses of portable media formats are blatant, others are a result of legal vagaries or “nuances of 
legality.” In other words, laws may not exist or may not be explicit about what users can do with downloadable 
programming once it is on their personal computers. In part, this is because every possible scenario cannot be 
accounted for or presupposed within the limits of the law. But manipulation of media on personal computers is 
generally private and the rest of society simply doesn’t know what one does in the privacy of his/her own home. 



364 




Until manipulated media is re-uploaded for public consumption, it is almost impossible to track illegal activity. 
However, once altered media are put on a networked server, it generally means that it is in the public domain and 
subject to any laws that may exist. 

Some of the more common alterations and creations of pod/vodcasts that occur, but are not legal, include the 
following; 

• The use of soundbytes or copywritten samples within a pod/vodcast without permission (i.e. the concept of 
remixing). 

• The assumption that all pod/vodcasting licensing agreements are the same and have the same restrictions, 
when in fact licenses have different stipulations and can only be used according to the particulars that are 
outlined therein. 

• The belief that obtaining permission to use a pod/vodcast only from it’s creator, but not from each 
performer or artist on the broadcast, is comprehensive authorization. Although a creator may be able to 
represent any artists on his or her pod/vodcast, this is not consistent with every performance. Although it’s 
not clear whether this is always necessary, it’s better to always err on the side of caution and obtain 
permission for use from all involved parties whenever possible. 

• The use of parts of a pod/vodcast in another pod/vodcast (or anything else) without permission. Simply 
because a broadcast may be free, you cannot take it apart to re-use it any more than you could re-use it in 
it’s entirety without permission (for profit or otherwise). 

• Redistribution of pod/vodcast without authorization. If a pod/vodcast or set of pod/vodcasts is removed 
from the web, a user who has downloaded this material cannot re-upload it for public consumption (free or 
otherwise). The difficulty with this is that even if a user wanted to ask permission to re-upload, if the 
material and/or the web host has disappeared (as does happen), who would s/he go to in order to ask 
permission? Although frustrating, redistribution of these media without consent is illegal by the letter of 
the law. 

• Creation and distribution of a derogatory or negative pod/vodcast. Since pod/vodcasting is not regulated 
by the government in the way that traditional radio and television are regulated by the Federal 
Communications Commission (FCC), anyone can generate and air a podcast or a vodcast. As a result, 
there is no topic that is off limits and no language that is too colorful. Although much of these media are 
free, they are still subject to legal laws of liable and defamation of character, just as for-profit programs are. 
Simply because these programs are not earning a profit, doesn’t mean that they aren’t accountable under 
the same laws. 

Perhaps the most interesting aspect about portable media and peer-to-peer (P2P) file sharing is not the legality, as 
P2P is legal in the letter of the law, but the ethical implications. As we know from recent and consistent media 
coverage of this practice, most peer-to-peer file sharers exchange popular, copy written music because it is an 
inexpensive way to obtain it. People seem to insist on sharing illegal material despite industry and government 
guidelines, as well as some harsh consequences. Why does this sort of behavior continue when it has been 
well-publicized that it is not acceptable within the limits of our laws? Is it purely and issue of money? 

Fortunately, many news, television, and radio programs, as well as independent artists, have become savvy to the 
popularity of free, downloadable, portable information and entertainment. Recently, media outlets have been 
creating and uploading regular audio or audio/video programs that are often free and always legal to download from 
the host websites. While having mainstream media create programming that is available for free is a move toward 
lessening illegal usage, it still does not explain the persistence of society to acquire for-profit resources dishonestly. 

One solution to this dilemma may exist with the approaches and ideas of Scott Sigler, podcast and print author. In 
an interview with Elisabeth Lewin (2008), Sigler states that print publishers at least, should embrace free, 
downloadable media rather than trying to compete with it. Instead of attempting to force their audience to purchase 
what the public clearly knows they can acquire for free, publishers/creators should think of novel ways to gamer a 
market. In a radical move, Sigler proposes that in the case of books, publishers should consider giving the main 
work away for free, via an audio or print download, in order to let the public get to know and like the author. Then, 
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publishers could have extensions, follow-ups, character details, etc. available for online purchase once they’ve 
captivated the audience. While this approach may run entirely counter to everything that publishers and marketing 
companies have practiced until now, it may be the way to both honor the idea and practice of free, downloadable 
entertainment, and yet provide some sort of profit base for the industry. It may also even encourage more legal 
attainment and use of portable media. 

Unfortunately, not only will most users remain ignorant of some of the laws surrounding pod/vodcasting simply 
because they can’t keep up, but lawmakers themselves will have difficulty in creating and upholding regulations. 

In reality, fair and honest use of portable media may be an issue of principles and morals, rather than strict legal 
structures. Technology itself rarely creates societal problems. Rather, it’s the users of technologies who 
manufacture problems by inappropriate uses and unethical choices. If society can be thoroughly educated to enjoy 
and justly share intellectual and artistic digital work, then individual members of society will perhaps be more 
inclined to monitor their own personal usage and the need for up-to-the minute, constantly changing laws may be 
lessened. At the time of this writing, many colleges and universities offer courses and even programs of study on 
ethics and ethical issues. In these programs, cyber-ethics and techno-ethics are becoming increasingly important 
elements. While these areas are relatively new disciplines of study, their focus is to address issues exactly like the 
ones being discussed here. 



Conclusion 

Given that it is difficult to predict the laws needed to address digital aspects of a society, laws regarding technology 
can only be, for the most part, reactive to situations that develop with or from emerging technologies (rather than 
proactive in order to prevent problems). Therefore, it may well be an issue of personal accountability and honesty 
with regard to appropriate use of portable media like podcasting and vodcasting, rather than an issue of regulations 
and consequences. It is an argument that is reminiscent of the tale of the chicken and the egg - which comes first, 
the laws that guide human behavior, or the behavior that necessitates the creation and implementation of laws? 
Whichever way it works, the legal and ethical issues surrounding portable media and portable media devices are 
constantly changing and will continue to change at a whirlwind pace. We may never institute a legal system that 
will consistently help society to remain honest when it comes to digital media, but we can make strides in 
developing an ethical and just population with which this type of media finds its audience. 



References 

Belanger, Y. (2005, June). Duke University iPod First Year Experience Final Evaluation Report. Retrieved October 
17, 2008 from <http://cit.duke.edu/pdf/ipod_initiative_04_05.pdf>. 

Boulos, MNK., Maramba, I., & Wheeler, S. (2005). Wikis, blogs and podcasts: a new generation of Web-based tools 
for virtual collaborative clinical practice and education [Electronic version]. BMC Medical Education, 
b(41). 

Campbell, G. (2005, November/December). There's something in the air: Podcasting in education. Educause 
Review, 40, 33-46. Retrieved on October 17, 2008 from 
<http://www.educause.edu/ir/library/pdf/erm0561.pdf>. 

Cebeci, Z., & Tekdal, M. (2006). Using Podcasts as Audio Learning Objects [Electronic version]. Interdisciplinary 
Journal of Knowledge and Learning Objects, 2. 

Chan, A. & Lee, MJW (2005) An MP3 a day keeps the worries away: Exploring the use of podcasting to address 

preconceptions and alleviate pre-class anxiety amongst undergraduate information technology students. In: 
Dirk HR Spennemann & Leslie Burr (eds). Good Practice in Practice. Proceedings of the Student 
Experience Conference 5-7th September ’05. Wagga Wagga, NSW: Charles Sturt University. 59-71. 

Chinnery, G. M. (2006). Emerging Technologies - Going to the MALL: Mobile Assisted Language Learning 
[Electronic version]. Language Learning & Technology, 70(1), 9-16. 

Creative Commons (2008). Creative Commons Attribution 2.5. Retrieved October 17, 2008 from 
http://creativecommons.Org/licenses/by/2.5. 



366 




Gardner, H. (1983). Frames of Mind: The Theory of Multiple Intelligence. Basic Books, New York, NY. 

Donnelly, K. M., & Berge, Z. L. (2006). Podcasting: Co-opting MP3 Players for Education and Training Purposes. 
Online Journal of Distance Learning Administration, 11(3). Retrieved October 17, 2008 from 
<http://www.westga.edu/~distance/ojdla/fall93/donnelly93.htm>. 

Godwin-Jones, R. (2005). Emerging Technologies - Skype and Podcasting: Disruptive Technologies for Language 
Learning [Electronic version]. Language Learning & Technology, 9(3), 9-12. 

Herkenhoff, L. M. (2006). Podcasting and VODcasting as Supplementary Tools in Management Training and 
Learning. Retrieved October 17, 2008 from <http://www.iamb.net/CD/CD06/MS/71_Herkenhoff.pdf>. 

Hiirst, W. & Waizenegger, W (2006). An overview of different approaches for lecture casting. Proceedings oflADIS 
International Conference on Mobile Learning 2006, July 2006. 

Ketterl, M., Mertens, R., & Morisse, K. (2006). Alternative content distribution channels for mobile devices. 

Retrieved October 17, 2008 from <http://www2.informatik.uni-osnabrueck.de/papers_pdf/2006_02.pdf>. 

Matthews, K. (2006). Research Into Podcasting Technology Including Current and Possible Future Uses. Retrieved 
October 17, 2008 from <http://mms.ecs.soton.ac.uk/2007/papers/32.pdf>. 

Meng, P. (2005). Podcasting & vodcasting: A white paper: Definitions, discussions & implications. Columbia: 
University of Missouri, lAT Services. Retrieved October 17, 2008 from 

<http://edmarketing.apple.com/adcinstitute/wp-content/Missouri_Podcasting_White_Paper.pdf>. 

Read, B. (2005). Seriously, iPods Are Educational [Electronic version]. The Chronicle of Higher Education, 57(28). 

Schnackenberg, H.L. & Vega, E.S., & Warner, Z.B. (2008). Podcasting and Vodcasting in Education and Training. 
In R. Luppicini & R. Adell (Eds.), Handbook of Research on Technoethics (pp. 668-679). Hershey, PA: IGl 
Global. 

Lewin, E. (August 22, 2008). Scott Sigler Talks About Social Media, Podcast Books And The Future Of 
Publishing. Retrived October 17, 2008 from 

<http://www.podcastingnews.eom/2008/08/22/scott-sigler-talks-about-social-media-podcast-books-and-the 

-future-of-publishing>. 

Tavales, S., & Skevoulis, S. (2006). Podcasts: Changing the Face of e-Learning. Retrieved October 17, 2008 from 
<http://wwl.ucmss.eom/books/LES/CSREA2006/SER4351.pdf>. 



367 




Better teaching methods for Teacher Education: Blackboard discussions 

improve critical thinking 



Zsuzsanna Szabo, Ph. D. 

Southern Illinois University at Edwardsville 

Jonathan Schwartz, Ph. D. 

University of Hawaii West O'ahu 

ZSUZSANNA SZABO is Assistant Professor in the Educational Leadership Department 

at the Southern Illinois University at Edwardsville. Her research interests include human cognition, team learning, 
classroom assessment, and gender issues in education. 

JONATHAN SCHWARTZ is Assistant Professor in the Division of Elementary Education at the University of 
Hawaii at West O'ahu. His research interests include inquiry, curriculum and instruction, and human cognition. 

Eor comments and any questions please contact Dr. Szabo at e-mail: zsuzsanna25 @ gmail.com 

Key words: critical thinking, online discussions, teaching methods 
Abstract 

An increasing number of colleges are incorporating online learning experiences into teacher education 
programs. As online experiences become a more common instructional tool, research is needed to examine their 
impact on student learning. Results from this study show that the use of Blackboard discussion forums as 
supplementary instructional tool in a face-to-face course, improved undergraduate preservice teachers abilities to 
demonstrate critical thinking skills. Participants in the study were 93 students registered in four sections of an 
Educational Psychology course at a Midwestern university. All participants in the study took the Canfield’s 
Learning Style Inventory as pre-test, to control for learning preferences; and all also responded to the Ennis-Weir 
test of Critical Thinking as pre- and post-test. In two of the four sections, as part of their coursework, students 
completed weekly Blackboard discussions on course topics, moderated by the teacher. Quantitative analysis from 
the critical thinking measure, and the analysis of online postings showed that preservice teachers developed their 
ability to analyze, synthesize and evaluate course content, as well as an increase in critical thinking abilities. 
Quantitative and qualitative analysis is presented along with implications for education. 

Critical thinking and higher thinking levels 

Critical thinking is often discussed in relation to skills such as logical reasoning, analyzing arguments, 
testing hypotheses, making decisions, estimating likelihoods, and creative thinking (Hallet, 1984; Ruggiero, 1975; 
Walters, 1994). Almost half a century ago Ennis (1962) presented a critical thinker as being characterized by her 
mastery of analytical operations that enabled her to judge relationships between propositions, evaluate, and defend 
beliefs. In defining critical thinking, a distinction has been made between the process and the product of thinking. 
Some theorists (Chance, 1986; Nickerson, 1984; Nickerson, Perkins, & Smith, 1985; Sternberg & Kastoor, 1986) in 
the process of explaining critical thinking use the model that involves a hierarchy of learning applied to the skills 
involved in the classroom teaching to encourage students to “progress” to higher thinking levels. 

Critical-thinking skills are often referred to as higher order cognitive skills to differentiate them from 
simpler (i.e., lower order) thinking skills (Halpern, 1998). Higher order thinking skills are relatively complex and 
require judgment, analysis, synthesis, and creativity; they are distinct from the skills that use memory processes as 
base and are applied in a rote or mechanical manner. Higher order thinking is reflective thought, sensitive to the 
context, and self-monitored. Bloom’s revised Taxonomy provides a useful tool to define critical thinking skills 
(Anderson and Krathwohl, 2001; Bloom, Krathwohl, & Masia, 1956). Higher levels of learning imply the use of 
critical thinking skills, metacognition, and the ability to analyze, evaluate, and develop new ideas. All these abilities 
in student learning are represented in the higher levels of Bloom’s taxonomy as revised by Anderson, Krathwohl, 
Airasian, Cruikshank, Mayer, Pintrich, Raths, & Wittrock, (2001). Bloom in the original taxonomy (Bloom, 
Kratwohl, and Masia, 1956) categorized the thinking process in six levels: knowledge, comprehension, application. 



368 



analysis, synthesis, and evaluation. But later Anderson, Krathwohl, et al. (2001) developed a two dimensional 
taxonomy: cognitive processes dimension, and knowledge dimension. The cognitive processes dimension comprises 
the six thinking levels which are similar to the original levels of Bloom’s taxonomy. One change was that action 
verbs were used instead of nouns, to stress the process of thinking and not as much the result of the thinking: 
Remembering, understanding, applying, analyzing, evaluating, and creating (Anderson & Krathwohl, 2001, pp. 67- 
68). The other dimension “Knowledge dimension” has four levels: factual, conceptual, procedural, and 
metacognitive. The two dimensional model offers a grid which is in fact a useful tool to plan teaching objectives and 
also to test students’ learning at the different levels represented in the taxonomy. Using the revised Bloom’s 
taxonomy teachers can measure if students’ performance implies higher order thinking. When students use critical 
thinking they use metacognition, which is an important level on Bloom’s taxonomy. Students need not only to 
understand the concepts they learn, and not only to be able to apply them (processes at lower levels of Bloom 
taxonomy); in order to reach higher levels of thinking (the upper levels in Bloom’s taxonomy) it is very important 
that students be able to analyze the learned content, to evaluate, compare and contrast the content they learn, and be 
able to create new ideas that have application in their practice. 

Critical thought implies the use of metacognitive thinking. Metacognition is the process of thinking about 
one’s own thinking (Matlin, 2006). Through the process of metacognition, students learn to evaluate their level of 
knowledge, reflect on the content they have learned, and become aware of necessary revisions on the respective 
content. The process of metacognition, the self-evaluation of own thinking, is part of the critical thinking skills that 
students need to employ during the process of learning. Bloom considered the metacognitive process among the 
higher level thinking processes in his taxonomy. 

According to Piaget (Piaget & Inhelder, 1973), metacognitive abilities are developed in formal operation 
stage of cognitive development and are important for the learning process. Through metacognition the student is 
able to monitor the learning process, adapt, and make necessary changes. Metacognition makes learners more aware 
“when they need to check for errors, why they fail to comprehend, and how they need to redirect their efforts." 
(Ertmer & Newby, 1996). This awareness is implied also in critical thinking process. The necessity for higher levels 
of thinking brings the need for targeted teaching methods that help students develop critical thinking, and improve 
higher order thinking skills. 



Critical thinking and teacher education 

Instruction for critical thinking is a central component of higher education curricula. The task facing 
college-level educators is to ensure that the teaching processes incorporate strategies that better reflect the rich 
complexity of critical thought. Teaching for critical thinking must be based on a more inclusive theoretical model of 
critical thinking that recognizes the multifunctionality, contextuality, and the emancipatory nature of thinking, and a 
sound pedagogical approach (Beyer 1997). Students need not only to understand the concepts they learn, it is very 
important that they analyze the learned content, evaluate, compare and contrast, and be able to create new ideas that 
have application in their practice. This is important especially in teacher education because classroom teachers need 
to be able to reflect on their practice in order to enhance teaching and learning. Reflective practitioners step back 
and examine classroom events that unfold. They analyze events and evaluate the success of their teaching and 
resulting learning. They then make instructional decisions to alter practice to make improvements. Reflective 
practitioners critically examine classroom events and make adjustments in order to maximize the effectiveness of 
meeting the needs of individual student learners. This is an essential part of teaching and must be conveyed to all pre 
service teachers. . 

Elder and Paul (2002) mention that to be skilled in critical thinking, a person needs to be able to take one’s 
thinking apart systematically, to analyze each part, assess it for quality, and then improve it. This is also important to 
create self-directed learners. 

Teaching preservice teachers to improve their critical thinking skills helps them become better thinkers and develop 
the ability to synthesize and analyze information, identify main ideas, cite evidence in support of a conclusion, and 
develop evaluation skills; all skills at higher levels of Bloom’s taxonomy (Bloom, Krathwohl, & Masia, 1956). 
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Biggs (1987, 1998, 1999) described two distinct approaches to learning revolving around distinct groups of 
learners: “Deep learning” used by students who are highly engaged and learn for the sake of knowledge acquisition, 
they study to learn and are motivated to go beyond the basic requirements. Deep learners use higher order cognitive 
skills, and critical thinking. The second approach to learning is called by Biggs “Surface learning” and students 
using this approach learn as much as it is necessary for passing, only the necessary to gain a passing grade or 
qualification. Surface approach learners are less cognitively engaged than their counterparts who are deep learners. 

It can be drawn the idea that deep learning involves critical thinking and in consequence it takes place at higher 
levels according to Bloom’s taxonomy. In consequence, assessing critical thinking should be part of the teaching 
process at college level to ensure deep learning. 

Teaching for critical thinking must be based on an inclusive theoretical model of critical thinking that 
recognizes the multifunctionality, contextuality, and the emancipatory nature of thinking, and a sound pedagogical 
approach (Beyer 1997). 

Some authors (Bigge & Shermis, 1992; Mayer, 1992; Swan, & Shea, 2005) recommend the use of teaching 
methods that require active student involvement where students apply higher order thinking skills in multiple 
settings. Technology offers such a tool to accomplish this goal. Technology tools used for instruction such as online 
threaded discussions enhance the process of teaching and learning by offering students an opportunity to 
communicate thoughts and develop understandings. Derry, Hmelo-Silver, Nagarajan, Chernobilsky, & Beitzel 
(2006), demonstrated the effectiveness of use of technology for teaching college courses. They suggest that 
technology integrated into the teaching and learning process helps students develop critical thinking skills. 

Use of technology to improve students’ thinking 

Teaching is a process of creating meaningful learning experiences, but teaching should not be limited to 
classroom activities. At the present one of the most prominent environments of outside of classroom teaching is the 
cyberspace (Coppola and Thomas, 2000). Technology use becomes one of the usual means of teaching at all levels 
of education. Since technology and worldwide communication have created a need to be prepared to think and work 
smarter, students need mental flexibility, they need to go beyond knowledge and understanding how something is 
done, they need to be able to apply their knowledge in multiple and creative settings. 

McFarlane (1997) stressed that when the use of technology in education is discussed, the most important 
issue for debate should be determining the purpose of technology use. McFarlane writes that there are advantages as 
well as limitation in the use of technology, for this reason the most important issue is related to how technology is 
used in education: “Computer use alone, without clear objectives and well-designed tasks, is of little intrinsic value.” 
(McFarlane, 1997, pp. 35). It is important that the use of technology to be purposeful and related to the content of 
teaching in order to be effective. 

Lei and Zhao (2007) note similar conclusions. They write that the purpose for what technology is used for is more 
important than the simple use of technology. These researchers examine how the quantity and quality of technology 
use affect student learning outcomes. They suggest that the quantity of technology use alone is not critical to student 
learning. “How much” matters when “how” is identified. For example, the simple implementation of technology 
(even if it is in large quantity - responding to the question “how much”), in the process of teaching and learning, 
does not guarantee the achievement of higher thinking skills. Heavy use of technology does not improve student 
performance; if the purpose and modality of technology use, does not have as purpose to improve the learning 
process. Instead, the importance of technology use in education responds to the question “how” technology is used; 
specifically, what are the goals for what technology is used in education. Moreover, the authors remark, “when the 
quality of technology use is not ensured, more time on computers may cause more harm than benefit.” (Lei and 
Zhao, 2007, pp. 286). 

Similarly, Koehler, Mishra, and Yahya (2007) state that the use of technology must be purposeful in order 
to serve higher levels of learning through improved pedagogies. They write that, “effective technology integration 
for teaching subject matter requires knowledge not just of content, technology and pedagogy, but also of their 
relationship to each other (pp. 746).” They present the use of technology in teaching as being part of the model of 
Technological Pedagogical Content Knowledge (TPCK). This model includes four components: Technology (T), 
that encompasses standard technologies used in educational setting; Pedagogy (P), includes the process and practice 
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or methods of teaching and learning; Content fCj, or the subject matter that is to be learned/taught; and Knowledge 
(K), the information base acquired by the student. Koehler, Mishra, and Yahya (2007) stress that the TPCK model 
includes also the relationship between the components; content, pedagogy, and technology. The use of technology 
must be purposeful in order to serve higher levels of learning through improved pedagogies. The purposeful use of 
technology will improve learning. This idea is similar to the one stressed by Lei and Zhao (2007), concerning the 
importance of the way how technology is used in education. 

According to TPCK model the use of technology to deliver the content to be taught is a pedagogical 
method that, if used properly can improve the teaching and learning process. 

The use of technology should increase deep learning. As it was previously mentioned, in a deep approach to 
learning, the material is embraced and digested in the search for meaning. Surface learning employs the least amount 
of effort toward realizing the minimum required outcomes. Surface learners are motivated to complete the task 
rather than assimilate the learning (Biggs, 1999). Using technology that requires higher levels of thinking would 
help students develop a deep learning. 

Sivin-Kachala and Bialo (1993) presented the effect of use of technology in education in a synthesis based 
on 86 research reviews. They show that the use of technology in teaching demonstrated a significant positive effect 
on achievement; has positive effects on student attitudes toward learning and on student self-concept. But along with 
effects on students, technology has influence on teachers as well. Teachers develop more student-centered teaching 
when using technology, and the student-to-student and student-to-teacher interaction shows an increase when 
technology is used in education. The authors mention that “it is not the technology that makes the difference but 
rather how teachers adapt and apply technology that makes the difference” (pp. 389). 

The most important issue in the use of technology in education is concerned with the purpose technology is 
used. Technology should be used in teacher education for at least two reasons; to improve content knowledge and to 
improve skills related to the use of technology. Teacher education should provide skills that are compatible with the 
teacher career in the age of technology. Also teacher education by using integrated technology as teaching methods 
should help preservice teachers develop higher order thinking skills (Hmelo-Silver, 2006; Ukpokodu, 2000). 

In the learning process as mentioned by Garrison and Anderson (2003) a community of inquiry integrates 
cognitive, social, and teaching elements that are not limited to social exchanges and are more than low-level 
cognitive interaction. Several researchers (Garrison, Anderson, and Archer 2000; Meyer 2003; Pawan et al. 2003) 
show that a community of inquiry is the integration of cognitive, social, and teaching presence. Garrison and 
Cleveland-Innes, (2005) talk about the fact that the “quantity of interaction does not reflect the quality of discourse” 
(pp. 135). On a similar line of thought Roybler (2002) found that voluntary and required message posting that were 
pertinent to the purpose of the discussion created higher student engagement. Roblyer and Wiencke (2003) show 
that consistent interaction in courses that use technology is associated with higher achievement and student 
satisfaction. 

Schumm, Webb, Turek, Jones, and Ballard (2006) in a study that compared face-to-face and online courses 
evaluating the level of critical thinking, found that the use of online discussions increased students’ critical thinking 
skills. They also stated that students had more complex questions, and increased contact in online format. 

The purpose of the present study was to examine if pedagogical methods used in teaching preservice 
teachers that imply the use of technology have as result the improvement of critical thinking. Based on the above 
mentioned research literature, two hypothesis were tested; (1) the use of Blackboard discussion forums as 
supplementary instructional tool in a face-to-face course will improve undergraduate preservice teachers’ critical 
thinking skills; and (2) an increase in preservice teachers’ critical thinking and deep learning will be demonstrated 
through the level of postings to the Blackboard discussion forums. 

Methods 

Participants 

Participants were 93 undergraduate students (82% were preservice teachers) in four sections of an 
Educational Psychology course at a Midwestern university (two in the Spring of 2006 and two in Spring 2007, see 
Table 1.). The sections were identical with respect to length, objectives, requirements, assignments, examinations. 
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and grading criteria. All four sections used the same text book, the syllabus for each section followed the same 
calendar of topics, and the content of study was the same for each week. Sections were taught by two different 
teachers, but with equal teaching experience. Two sections were considered the “traditional” groups (21 students in 
2006 and respective 22 in 2007; with 67% respective 77% female students). Traditional teaching methods were 
lectures, in-class discussions, homework assignments, and in-class comprehensive test as final examination. The 
other two sections (27 students in 2006 and respective 23 in 2007; with 74% respective 65% female students) were 
considered the “technology” group. 



Table 1. Participants in the study by gender and type of teaching method 



Semester/ 
teaching method 


Eemales 


Males 


Total participants 


2006/ traditional 


14 


7 


21 


2006/ technology 


20 


7 


27 


2007/ traditional 


17 


5 


22 


2007/ technology 


15 


8 


23 


Total 


66 


27 


93 



As part of their coursework, students in the “technology” group participated in both, in-class and online 
activities. The in-class activities were the same as for the traditional group: small group in-class discussions, 
homework assignments, and in-class comprehensive test as final examination. For the online activities students were 
required to post minimum 6 times per semester (half number of topics studied) reflections on the topics studied 
across the semester using the Blackboard discussion board, and respond to other students’ postings. The Discussion 
Board postings were moderated by the teacher. Students were presented with Grading Rubrics for each assignment 
as well as for Blackboard postings (see Appendix 1). The purpose of online discussion board postings was to 
provide students with opportunities for discussions and reflections on the content. The guidelines that students 
received through the rubric encouraged that their reflections posted on the discussion board were proof of higher 
order thinking. The teacher moderated the discussion board postings more in the first couple of weeks into the 
semester, so that students were encouraged to use higher order thinking in their reflections and responses to other 
postings. Later in the semester the teacher only responded to eventual questions or made clarifications. 

Students in the “traditional” group, using the same textbook and syllabus, participated in the same in-class 
activities, had the same homework assignments and final examination, and followed the same curriculum across the 
semester, less the assigned Blackboard discussion board postings. However to compensate the difference students in 
the “traditional” group were asked for homework assignment to complete short reflections on the topics studied. 
Students in the “traditional” group were provided with the same grading rubrics as the students in the “technology” 
group. 



In the Spring of 2006 two sections of the same Educational Psychology course were randomly assigned, 
one to the “traditional” teaching style and the other one to the “technology” teaching style described above. In 
Spring 2007 teacher’s used teaching style was switched, so the teacher who taught in the Spring 2006 using 
traditional methods now used in addition the technology as represented through the Blackboard discussion board 
postings, and vice versa, the one who taught the “technology” section in Spring 2006, now was using traditional 
methods (no Blackboard discussion board postings). 

All participants in the study took the Canfield’s Learning Style Inventory (Canfield, 1992) first week in the 
semester to control for any learning environment predispositions among the students. Students also took the Ennis- 
Weir Test of Critical Thinking (Ennis & Weir, 1985) during first and last week in the semester (as pre- and post- 
test), with the purpose to observe any changes in their critical thinking in the process of learning over the respective 
semesters. 

In addition, only the students in the technology group were also asked to complete an “End of semester 
feedback” form (see Appendix 2) concerning their course experience, and the one related to using Blackboard 
reflection posted on the Discussion board. 



372 




For this study purposes students were assigned a code. This code was created from a first three digit 
number representing the group (section of class and semester), followed by a two digit number representing an order 
number for each student, and next a three digit number representing the topic order across the semester and the 
number of postings per topic. 

The pre- and post-test Ennis-Weir Tests of Critical Thinking (EWCT) were double blind scored to 
condition and time of assessment (inter rater r =.79), then analyzed to observe changes in critical thinking between 
students in the four sections. There were two raters, the first, one of the teachers who taught the students involved in 
this research; and the second rater was a faculty teaching the same content but from another university, and was not 
involved in teaching the students in this research. This was chosen intentionally to control for rater bias. 

Blackboard discussion postings were analyzed using a rubric (see Appendix 3) that was developed based on 
Bloom’s modified taxonomy (Anderson and Krathwohl, 2001) measuring for factual, conceptual, procedural, and 
metacognitive levels of postings. Postings at the higher levels of Bloom’s taxonomy corresponding to higher levels 
of thinking, deep level of learning and use of critical thinking. All Blackboard discussion postings on the class 
forums (appropriate topics from the syllabus) were scored separate by two raters (inter-rater reliability r =.82) using 
the above mentioned rubric. Quantitative analysis was performed comparing changes in Blackboard discussion 
posting levels for individuals across the semester. 

Measures 

Canfield’s Learning Style Inventory (CLSI) determines which learning environments and which types of 
instructors are best for particular students (Canfield, 1992). Participants taking CLSI respond to a 30 questions 
inventory by ranking each response to questions on a scale of 1 to 4. The CLSI is designed to determine which 
learning environments - and which instructors -are best for particular students. Scores were calculated for 
Conditions (i.e., teamwork, independent study, competition, classroom discipline, organized coursework, a close 
relationship with the instructor, or detailed information on assignments and requirements). Content , Mode (i.e., 
through listening, reading, interpreting illustrations or graphs, or through hands-on experience), and Expected 
Performance (how well does the student expect to perform in the class?) related to general course taking 
experiences. 

Ennis-Weir tests of Critical Thinking (EWCT) was developed (Ennis& Weir, 1985) to help evaluate a 
person's critical thinking ability of writing a critical argument to a specific situation. Participants are required to 
respond in writing to an eight paragraph fictitious letter written by a “concerned citizen” to a journal editor in 
regards to night parking on streets. The writer of the letter presents 8 specific reasons. Respondents need to present 
their logical and critical reasoning for each of the eight points (showing their reasoning in agreement or 
disagreement, and logical thinking about the arguments from the fictitious letter), and lastly give a general comment. 
Scoring is done using the specially designed scoring rubric provided with the test manual. Scores can be obtained for 
each of the 8 points and total scores. 



Results 

The principal purpose of this study was to assess if the use of Blackboard discussion forums as 
supplementary instructional tool in a face-to-face course will improve undergraduate preservice teachers’ critical 
thinking skills; and if an increase in preservice teachers’ critical thinking and deep learning will be demonstrated 
through the level of postings to the Blackboard discussion forums across the length of a semester course. 

Results from the comparison across groups and semesters shows that there were no statistically significant 
differences in the learning styles as measured by Canfield’s Learning Style Inventory between students in all groups 
(t = 1.67, df = 91, p = .098; Cohen’s d= .068). 

Results from the Ennis-Weir test of critical thinking show that there was a statistically significant increase 
in critical thinking skills as presented by students in the technology groups, but not for the students who were in the 
traditional teaching groups (Table 2). 
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Table 2. Descriptive statistics by groups for results on Ennis-Weir Tests of Critical thinking 



Semester/ group 


Number of subjects 


Mean 


SD 


Spring 2006 - technology 


27 


18.59 


3.33 


Spring 2006 - traditional 


21 


10.19 


4.95 


Spring 2007 - technology 


23 


21.35 


2.62 


Spring 2007 - traditional 


22 


11.64 


5.45 



At pre-test time on Ennis-Weir Tests of Critical Thinking there were no statistically significant differences 
in critical thinking between participants in the traditional and the technology group (E 3, 89 = .390; p=.76). This 
means that all groups were comparable, and students’ abilities for critical thinking were comparable at the beginning 
of each semester as well as across groups and semesters. At post-test there were statistically significant differences 
in critical thinking abilities between technology and traditional groups (E 3, 89 = 37.46; p=.0001). 

Table 3. ANOVA at pre-test and post- test for groups involved in the study 







Sum of 
squares 


df 


MS 


E 


Sig. 


Pre-test 


Between groups 


24.98 


3 


8.38 


.39 


.760 




Within groups 


1897.94 


89 


21.33 








Total 


1922.93 


92 








Post-test 


Between groups 


1960.21 


3 


653.40 


37.47 


.001 




Within groups 


1552.06 


89 


17.44 








Total 


3512.28 


92 









The effect size was large in both semesters: Spring 2006 Cohen’s d= .71; Spring 2007 Cohen’s d = .75. 
Statistically significant change in critical thinking from pre to post-test was found only for participants in the 
technology groups (t=15.04, df=49, p=.001). 

The analysis of Balckboard postings (using the scoring rubric in Appendix 3), over the time of a semester 
show an increase in student performance level and use of higher order thinking (revised Bloom’s taxonomy; 
Anderson and Krathwohl, 2001). Across the semester students’ online postings showed statistically significant 
increases in levels of application, analysis, evaluation, and creation, for conceptual, procedural, and metacognitive 
levels (results in Table 4 below). 



Table 4. Quantitative results from Blackboard discussion postings 







Sum of 
squares 


df 


MS 


E 


Sig. 


Eactual 


Between groups 


147.48 


11 


13.41 


4.94 


.001 




Within groups 


844.74 


311 


2.72 








Total 


992.22 


322 








Conceptual 


Between groups 


187.39 


11 


14.04 


6.02 


.001 




Within groups 


879.77 


311 


2.83 








Total 


1067.16 


322 








Procedural 


Between groups 


425.17 


11 


38.65 


11.41 


.001 




Within groups 


1053.09 


311 


3.39 








Total 


1478.27 


322 








Metacognitive 


Between groups 


240.69 


11 


21.88 


7.06 


.001 




Within groups 


963.96 


311 


3.1 








Total 


1204.66 


322 
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Discussions 



The results from Canfield Learning Style Inventory imply that students at the beginning of the semesters 
were homogeneous in what concerns learning and teaching style preferences; suggesting that students’ preferences 
for specific learning environments possibly did not affect their levels of critical thinking. These results are very 
important because they show that students in all four groups in study were equal in what concerns preference of 
learning in a course. 

Results from the Ennis-Weir critical thinking test show that only students in the “technology” groups 
presented an increase in critical thinking skills across the semesters in the study. Considering that the content to be 
learned, the textbook, and the syllabus calendar was the same for all groups this implies that the teaching methods 
used for the technology group (Blackboard discussions posting) helped students increase their critical thinking 
abilities. Given that each of the two technology groups was taught by a different teacher, the results are more robust 
and support the hypothesis that the difference in critical thinking among students is due to the reflections posted to 
the Blackboard discussion forum. 

Blackboard discussions required students to post reflections on specific topics and to respond to other 
reflections. Their reflections implied mastery of the topic, proof of ability to apply to content to real life situations, 
analysis and comparison of different theories learned, and development of new ideas (creativity). All these are also 
demonstrations of deep level of learning and use of critical thinking. Students from the “traditional” group were 
asked to show their reflection on the topics in study through the homework assignments. But they did not have the 
opportunity to read each other’s reflections as the “technology” group students had (due to the Blackboard open 
postings). We suspect that the differentiated results come from the opportunity for discussion involved in 
Blackboard discussion board postings. 

The above results support our first hypothesis that the use of Blackboard discussion forums as 
supplementary instructional tool in a face-to-face course will improve undergraduate preservice teachers’ critical 
thinking skills. Our results support also the literature mentioned by Elder and Paul (2002) and demonstrations from 
Derry, Hmelo-Silver, Nagarajan, Chernobilsky, & Beitzel (2006), concerning the effectiveness of use of technology 
for teaching college courses. 

In what concerns our second hypothesis that an increase in preservice teachers’ critical thinking and deep 
learning will be demonstrated through the level of postings to the Blackboard discussion forums a qualitative 
analysis of postings was conducted. Erom qualitative analysis point of view across the semester Blackboard 
discussion board postings were better developed and presenting higher levels of thinking, also the number of 
postings categorized at higher levels of Bloom taxonomy increased overall across the length of the semester, as 
shown above in the quantitative analysis of the same postings (using Bloom’s revised taxonomy as scoring rubric). 
Results from this research support what Biggs (1987, 1998, 1999), Koehler, Mishra, and Yahya (2007), and other 
authors (Bigge & Shermis, 1992; Mayer, 1992; Swan, & Shea, 2005) present concerning the use of technology as a 
teaching method as purposeful in order to serve higher levels of learning through improved pedagogies. Use of 
technology in face-to-face courses as a supplementary method of teaching showed the same results as Schumm, 
Webb, Turek, Jones, and Ballard (2006) found that the use of online discussions increased students’ critical thinking 
skills. Blackboard discussions also helped students had more complex questions, and increased contact in online 
format across the semester as compared to only face-to-face teaching methods. 

Eor example postings for topics at the beginning of the semester were mostly repeating the content and 
asking questions at the level of understanding. 

Emily (names are changed for all examples for confidentiality purposes), is an example of a student who 
demonstrated increased ability to think critically over the course of the semester. Early postings by Emily showed 
thinking at lower levels of Bloom taxonomy such as factual-remembering, understanding and applying. Emily's 
writings consistently recalled facts about theories and theorists; 

“According to the text, critical periods are time spans that are optimal for the development of certain 
capacities of the brain. Critical periods coupled with optimal environments further cognitive development. Since it is 
possible for children to develop at different times, how is a teacher supposed to create optimal environments for all 
students ? ” 
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Later Emily makes attempts to analyze and relate the topic in study with previously studied topics: 

“[...] According to the textbook some factors influence self-efficacy: previous experiences, observing 
others, teacher’s comments, environmental and physiological aspects. From what we learned previously I think 
students with high motivation probably have also high self-efficacy. I also would relate self-efficacy with the way a 
person makes causal attributions. [...] Therefore, as future teachers, one of our goals should be to help increase 
self-efficacy in our students. I think I would be able to come up with some methods I could apply in classroom 
setting ” 

In later stages of the class, Emily showed more of a tendency to reflect on her own thinking. She 
demonstrated evidence of analysis, evaluation and, in one instance, creation. Below is Emily’s last posting: 

"Motivation is such a key component to learning. It is the drive behind children that makes them 
want to learn. Without a reason to want to learn, why should they? It doesn't always have to be a treat. In 
fact my biggest motivation as a high school student was my desire to grow up go to college and become a 
teacher. It is all about what makes the child value education. The hard part is that it [motivation] is 
different for each student. That is our job as teachers, we need to motivate and reach as many of our 
students as possible. But first of all I think we must get to know our students, and know what they value, 
and what motivates them. " 

Analysis of the excerpt shows how Emily started to make connections to previous experience by 
relating the concept of motivation to her desires as a high school student. She goes on to evaluate 
motivation and identify her role in the process. In her later postings, it was not uncommon for Emily to 
project forward and evaluate how student motivation would impact her teaching. 

Students’ reflections across the semester developed from simple report on the content learned and proof of 
understanding of the information, towards proof of reflective and critical thinking, application, and analysis of the 
content. They also show evaluation of the theoretical concepts: 

“/ think Piaget’s theory makes more sense than the Classical behaviorist theory. In the end we are thinking 
beings, we might be able to learn something from reflex but we also think about what we learn. ” 

Michael is another student who makes explicit connections in his postings between the course 
content and home life. 

"This is a subject that is very close to home for me right now. My son is completely unmotivated. I 
have tried punishment, encouragement, giving money but nothing works. He refuses to do his home work 
and would have straight A' s if not for home work. I liked the different approaches that we talked about, 
especially letting him come up with his own ideas. I am going to try this approach with him and see if 
letting him have ownership of what happens can help him. I hope something will work soon for him. Thank 
you all for the ideas. " 

The student demonstrates that he is able to apply and test just as in a mini-research in a real life, 
the theory learned in the course. There is demonstration of analysis and evaluation. Later to the following 
topic Michael shows again critical thinking, metacognitive abilities and creative thinking: 

“It seems that Operant Conditioning and Motivation do fit hand in hand, however, I feel that 
there is a difference. A student who does not study for an exam and does well, may not be inclined to study 
for the next test. ..But with motivation (to learn), it's definition is the "student's tendency to find academic 
activities meaningful and worthwhile and to try and get the intended learning benefits from 
them". ..Operant conditioning is voluntary and the behavior precedes the stimulus, with Motivation students 
are presented with a challenge and it's at that point they decide whether or not it's something they are 
interested in and if they want to learn the information. " 

Analysis of statements from the “End of semester feedback” (only from participants in the technology 
group) show that participants report better learning and feelings of enhanced performance due to team discussions, 
and enjoyed the weekly Blackboard discussion postings since they could share more information, reflections, and 
questions outside the classroom. 

“I have found in the beginning of the semester very difficult to write reflections on the topic of the week; by 
around midterm I felt more comfortable and now when I think back I really enjoyed the Blackboard discussions and 
I think I learned more through reflecting on the topics than by only reading the book and in class small groups 
discussion. ” 

Also they reported that the out of class discussion opportunity helped them understand and share the 
learned content, as well as think about applications to real life educational situations. 

“If I were to modify anything to this course I would want to have weekly reflections as requirement, not 
only a minimum of six. I think that even if they were a big effort in the beginning, we learned from each other and 
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hud a place to continue our discussions after class, especially when we had more ideas of how to apply a concept. ” 
[...] “I liked that we could share ideas, and help each other come up with a better understanding of the content” 
Some participants reported that before taking the course they had limited knowledge about “Blackboard,” 
and at the completion of their performance they learned not only what a discussion board is for, but they really 
enjoyed online discussions and planned on using the method in their future teaching. 

“This was my first semester to use Blackboard and I was really afraid that I will not be successful. But I 
felt comfortable sharing with everyone. I think we helped each other and by the end of semester I am glad that we 
used the Blackboard. ” [...] “Using Blackboard helped me develop new skills and I think I will be more inclined to 
use technology in my future teaching. ” 

The results of this study support Elder and Paul (2002), as well as results from Krentler and Willis-Flurry 
(2005) that the use of technology and online discussions increase student learning and critical thinking abilities. In 
the current study the results show that the use of Blackboard postings in form of weekly reflections and comments 
was the factor that improved students’ critical thinking skills. 

Despite the positive results from this study there are several limitations. Even if study sections were 
randomly assigned to the teaching methods used in the course, the entire research study took place in sections of the 
same course. Future research is needed to evaluate changes in critical thinking in other content areas, in other 
courses where teaching methods that involve online discussions are used along with face-to-face traditional teaching 
methods. Another limitation of this study was the small sample size (only four sections of the same course). For this 
reason the present research has a limited generalizability; only to Educational Psychology courses taught for 
undergraduate preservice teacher education students. 

Conclusions 

Results of this study show that teaching methods that used technology in the form of Blackboard 
Discussion Board increased critical thinking skills in preservice teachers and also an increase in preservice teachers’ 
critical thinking and deep learning demonstrated through the level of postings to the Blackboard discussion forums. 

Along with an increase in critical thinking the use of online discussions gives preservice teachers 
experience with the use of technology for teaching purposes. 

In the new era when teachers must develop the same technology related skills as their future students, the 
teaching of future teachers must go beyond the theoretical content of human development, learning, and classroom 
assessment. In teaching the future teachers college faculty must prepare them to “walk the talk” and provide them 
with as many and diverse and high-quality classroom experiences as possible. Preservice teachers must learn skills 
pertaining to the new “multitasking” and technology savvy generation of students. Through the use of individual and 
team work, the implementation and use of technology, and asking preservice teacher education students to reflect, 
present, evaluate, and apply the content learned in new and creative modalities we help them also improve their 
critical thinking abilities. The results from this study show that blending technology along with face-to-face teaching 
will increase critical thinking among preservice teacher education students. The next question for a future research 
would be “What is the optimal combination of face-to-face and technology used to maximize student learning?” 
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Appendix 1 

Rubric for online discussions 

(presented to students as guideline for Discussion Board postings) 



Criteria 


Points 


Demonstrates an excellent understanding of key concepts; presents proof of critical 
thinking, ability of application, and evaluation of the theory, is able to compare and 
contrast the different topics learned across the semester; is able to evaluate the theory has 
personal reflection and comments to other postings; contributes in a timely and relevant 
manner; exceeds minimum number of required postings; writes clearly and logically; 
brings proof of information outside of the textbook 


18-20 


Demonstrates understanding of most key concepts, rarely has proof of critical thinking and 
application of the theory to practice; mostly shows agreement with other postings; does 
not bring consistent personal performance; generally contributes in a timely and relevant 
manner; meets minimum number of postings; generally writes clearly and logically 


15-17 


Demonstrates limited understanding of key concepts; contributes in a sporadic manner; 
short and rushed postings; only shows agreement or disagreement, does not contribute 
with personal reflection and comments; does not show critical thinking, and has no proof 
of examples for application of theory to practice. Has only the number of minimum 
required postings. 


12-14 


Demonstrates weak or no understanding of key concepts; Rarely participates freely; 
mostly “just to do the job” postings; short and irrelevant remarks. There is no proof of 
understanding the application of theory to practice. 


Less than 12 
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Appendix 2 



End of semester feedback 

Think about our classes - Educational Psychology - in this semester and please give your short response to the 
following questions: 

1 . What teaching method we used in this course was the most helpful for you in order to learn the material for 
this course? 



2. Did you find helpful for your learning process the assignment of posting reflections and comments to the 
Blackboard discussion board? 



3. What was something you did not find helpful? 



4. List a couple of ideas that would improve the class and which we as a class could use if we were to start all 
over again. 
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Appendix 3 



Scoring rubric for Blackboard discussion postings 



The cognitive process dimension 


The knowledge 
dimension 

Factual 

Knowledge 

The basic 
elements 
students must 
know to be 
acquainted with 
a discipline or 
solve problems 
in it. 

Conceptual 
Knowledge The 
interrelationships 
among the basic 
elements within 
a larger structure 
that enable them 
to function 
together. 

Procedural 

Knowledge 

How to do 
something, 
methods of 
inquiry, and 
criteria for using 
skills, 
algorithms, 
techniques and 
methods. 

Meta-cognitive 

knowledge 

Knowledge of 
cognition in 
general as well 
as awareness and 
knowledge of 
one’s own 
cognition. 


l.Rememb 

er Retrieve 
relevant 
knowledge 
from long- 
tern 

memory 


2. Understand 

Construct 
meaning from 
instructional 
messages, 
including oral, 
written, and 
graphic 

communication. 


3. Apply 
Carry out 
or use a 
procedure 
in a given 
situation 


4.Analyze 

Break 

material into 
constituent 
parts and 
determine 
how parts 
relate to one 
another and 
to an overall 
structure or 
purpose 


5. Evaluate 

Make 
judgments 
based on a 
criteria and 
standards 


b.Create 

Put elements 
together to 
form a 
coherent or 
functional 
whole; 
reorganize 
elements into a 
new pattern of 
structure. 



Anderson. L.W. (Ed.), Krathwohl, D. R. (Ed.), Airsian, P.W., Cruikshank, K.A., Mayer, R.E., Pintrich, P.R., Raths, 
T., & Wittrock, M.C. (2001). A taxonomy for learning, teaching and assessing: A revision of Bloom’s Taxonomy of 
Educational Objectives. Boston: Allyn & Bacon. 
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Efficient learning in serious games: a cognition-based design guidelines 

approach 



Erik D. van der Spek, Pieter Wouters, Herre van Oostendorp 
Utrecht University 

Abstract 

Games provide real opportunities for engaging students in learning educational material in contextually and 
perceptually rich environments. However, as much potential as there may be, until now there has been only limited 
evidence substantiating the claimed possibilities. More importantly, if positive results have been found, it remains 
unclear what has caused the increase in performance within the game. We therefore propose possible ways of 
enhancing serious games with the aid of research done in other fields of education and instructional design, in order 
to come to a set of cognition-based design guidelines for the development of effective serious games. 

Introduction 

Much already has been written on the potential of serious games for training and education, alternatively referred to 
as Games-based learning. Since the coinage of the term ‘serious games’ by the Serious Games Initiative in 2002, an 
exact search on Google Scholar already returns around 800 results in the last six years alone, and that naturally 
excludes the research done before 2002 and alternative wordings, in a field that’s active since the seventies. Many of 
these papers advocate the positive potential of serious games for education, for a number of reasons. For instance, 
virtual environments are environments and thus provide a rich context to a training, as compared to a textbook. 
Furthermore, because the education takes part in an interactive world, the player can form and test hypotheses, 
fostering cause-and-effect thinking. Because of this, games are not only suitable for information acquisition, but also 
knowledge construction, where learning involves building a mental representation (cf. Mayer, 2001). But above all, 
games are able to provide engaging ways of communicating the educational material to the player and traditional 
teaching equipment less so. Additionally, according to Prensky (2007), because you have to learn in order to 
progress in a game, for many digital natives playing games now has become a natural way to learn (and employ) 
knowledge. 

For all the interest surrounding the potential of serious games, only little is still known about the 
effectiveness thereof. In a review of all the contemporary literature on games-based learning, we found 28 scientific 
articles on serious games with any sort of experimental data, covering the whole gamut of different learning 
outcomes (Wouters, Van der Spek & Van Oostendorp, 2008). Most of these reported different degrees of positive 
results, with 58% of the learning goals met, 1/6* inconclusive and 1/4* reporting no increase in performance. This 
would indicate that, generally, serious games are sound instructional media. However, as Vogel et al. (2006) also 
noted in their meta-analysis, the literature is rife with methodological flaws, highly subjective or utilizing 
incomparable test strategies. Furthermore, with so little of the games’ efficiency reported on in scientific papers, and 
the development of serious games frequently being a costly and laborious effort, one can raise the suspicion of 
publication bias. 

While we concur with the idea that serious games are a better way to educate people in at least some areas 
of instructional material, this knowledge remains at a global level of games at best. One game may be better than a 
textbook, or one game may be better than another game, but the exact causes of this increase in performance are still 
unknown. In more scientific terms: when trying to ascertain the construct validity of the perceived increase in 
performance with games-based learning, it is still unclear which factors underlie the causal construct. Now that the 
scientific community has preached the gospel and shown that serious games can work, one of the major next 
challenges then becomes understanding why these games work, and, conversely, why other serious games fail to 
meet their intended learning goals. 
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The other side of the coin 



Among all the potential opportunities afforded by the use of serious games for training and education, the downsides 
are easily, and in fact often, overlooked. While games are not like real life, and therefore make it possible to train 
e.g. military personnel in situations that would otherwise be dangerous, games are also not like real life and, without 
fear of bodily injury, do not imbue a real sense of dread and handling dangerous situations accordingly into the 
player. 

Games, by their nature, have an edge over textbooks when it comes to presenting multimodal information; 
however some caution here is prudent, as the limitations of the system still make affective responses to the virtual 
environment different than to a real world situation. Standing inside a virtual theatre feels less impressive than 
standing in the theatre it tries to mimic (Houtkamp, Van der Spek & Toet, 2007). In part this may be because of a 
different context and feelings of expectancy, but also because of the lack of multimodal information such as smell, 
temperature, echoes, air currents and peripheral vision, something few commercial-of-the-shelf (COTS) games will 
ever accomplish. For the serious games developers it also doesn’t help that many entertainment games developers 
have vastly greater funds and development teams to create state of the art visuals, against which the serious games 
for education will look bleak in comparison. Because many people of the digital generation will have media 
schemata (IJsselsteijn, 2003) that are accustomed to highly photorealistic imagery, a less advanced game may not be 
adequately equipped to facilitate the needed suspension of disbelief. Additionally, up to 60 or 70% of people 
immersed in virtual environments may experience disorientation or other cybersickness related effects (Stanney & 
Kennedy, 1997), something that is also known to influence the affective appraisal of virtual environments (Van der 
Spek & Houtkamp, 2008) and which will, by extension, harm the transfer of training of the serious game if this 
relies on responding to affective situations (such as crisis management or coping with dangerous environments). 

The push towards increased realism will not only influence the affective component of games: with more 
happening simultaneously on screen, the cognitive system of the player will also be taxed more. We contend that 
one of the reasons why serious games are not realizing their full potential lies in the taxation of the cognitive system, 
and a lot of progress can be made if we are able to discover mechanics in game design that could ameliorate the 
cognitive load on the player of the serious game. By finding the underlying cognitive principles that not only reduce 
cognitive overload, but also improve learning itself, we will be able to turn serious games into more efficient 
learning environments. Given the relatively underwhelming results from serious games thus far, we therefore 
contend that this is a relevant research topic that deserves more attention. 

Adjusting the game to the player 

A similar call to the scientific community for finding out the underlying principles that constitute good serious game 
design, has been voiced by Wilson et al. (2008). In an extensive review of serious games literature, they focused on 
the relationships between game attributes and learning outcomes, where they distinguish between a large number of 
game attributes, such as fantasy, control and representation, in order to see which attributes would work best for a 
given learning outcome. Something similar surfaced in our review (Wouters et al, 2008) where we linked the 
cognitive and affective complexity of a game to possible learning outcomes. These are two broad ways of looking at 
possible causes for the effectiveness of serious games; here, we would like to focus more on specific design 
mechanics to enhance games-based learning. 

Different types of knowledge can be mediated by a serious game. Mayer (2001) distinguishes roughly 
between two views of learning: information acquisition and knowledge construction. Information acquisition sees 
learning as receiving information of an instructor; in the knowledge construction view the learner is actively creating 
a mental model, or internal mental representation of the instructional material (Moreno & Mayer, 2007). It is 
important to note that from hereon, we will primarily concern ourselves with the second view, and consider a mental 
model to be an internal representation of all the actors and events in a game world, their goals or possible effects, 
and how they relate to each other. 

That we focus on knowledge construction is because we think that a game, being an external representation 
of some world to be cognized, with actors and causal relationships, naturally generates mental model construction 
and is therefore better suited to this task than to information acquisition. Subsequently, we contend that in order to 
maximize the potential of learning with serious games, the serious game should be designed in such a way that 
knowledge construction occurs at an optimal level. To achieve this, the game should be better adjusted to the 
cognitive abilities of the player. 
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Evidence from learning with multimodal learning environments 

While the research on games-based didactics is still relatively in its infancy, a lot of scientific inquiry has already 
been done on other multimodal forms of learning, such as web-based learning and learning from animations. As 
games could, on a very basic level, be seen as interactive, (competitive) animations, especially the latter may 
provide insight into more effective ways of modeling instructional content. Animations too were expected by their 
nature to facilitate better mental model integration of complex material, particularly if the information was 
procedural, because comprehension would rely on our ability to mentally animate static images (Hegarty, 1992). 
Like games, however, there turned out to be scant evidence that substantiated that claim. In fact, in a review, 
Tversky, Morrison, and Betrancourt (2002) found that dynamic visualizations did not generally outperform static 
images; Hegarty, Kriz and Cate (2003) concluded the same after comparing both animations and static diagrams in a 
mechanical context with procedural information, although both did perform better than a single image. As a result, 
the research has since then focused on how to improve learning from animation, leading to a number of guidelines 
of instructional design (Van Oostendorp, Beijersbergen & Solaimani, 2008). We contend that some of these may 
also potentially benefit the instructional design of serious games, and therefore warrant a closer look. 

Mitigating cognitive overload 

One reason why games may not work as well as they could when it comes to games based learning, is that as a 
learning environment, they are quite complex. A lot of things will generally happen concurrently on-screen, while 
players have to navigate a virtual world in an unnatural manner with a gamepad or keyboard and mouse, at the same 
time remembering the rules of play and attending visual communications the game is providing, and simultaneously 
have to learn the educational material. It is unsurprising then that players may experience high amounts of 
extraneous cognitive load, which, because the human’s working memory has a limited capacity, will impede 
learning the relevant material (cf. Paas, Renkl & Sweller, 2003). Therefore, mitigating any unnecessary cognitive 
load, or extraneous cognitive load as it known in Cognitive Load Theory (cf. Van Merrienboer & Sweller, 2005), 
could help the serious game become a more effective teaching tool. 

One way to reduce the extraneous cognitive load of learning with multimedia is handled extensively by 
Mayer and Moreno (2003), and revolves around dual coding theory, or dual channel, and the split-attention effect. 
This states that we can process information concurrently both verbally and pictorially (dual channel), and if either of 
the channels is overloaded, it can be offloaded by providing the information in the other modality. So instead of 
presenting verbal (e.g. textual) information next to a diagram, which causes split attention, it would be better to 
provide the textual information audibly while the user looks at the diagram. 

With so much happening on-screen in a contemporary perceptually rich virtual world, many students will 
not know where to look, especially as irrelevant information can be highly salient, while relevant information 
relatively inconspicuous. Another way to mitigate extraneous cognitive load could subsequently be to reduce the 
amount of visual searching the player has to do — unless of course training to be more efficient in visual searches is 
the main goal of the game! In order to reduce this extraneous search, one could use visual or auditory cues to guide 
the student’s attention to the relevant educational material on-screen. A positive effect, albeit arguably weak, of 
cueing was found in Tabbers, Martens and Van Merrienboer (2004) in the context of a web-based multimedia lesson 
in instructional design. The effect of different forms of implicit and explicit cueing on path finding in a game 
environment was tested by Steiner and Voruganti (2004), leading to mixed results when it came to speed and 
accuracy, but they did find that experienced gamers were significantly more likely to notice implicit cues. 
Interestingly, cueing as a conscious way to mitigate cognitive load is starting to be picked up by the entertainment 
games industry as well, and will be used greatly in two soon to be released large productions. In EA/DICE’s 
Mirror’s Edge, the player takes the role of a parkour-style runner that delivers postal packages in a first person 
action game. Because heavy emphasis is placed on navigating through a detailed environment at high speeds while 
players have little time to look around, the objects a player can interact with have been colored bright red, which 
works as a cue to guide the player’s attention to the right direction. Somewhat more subtly, Microsoft Game 
Studios’ Fable 2 has the player accompanied by a personal pet dog. This intelligent agent will signal for danger or 
sniff at important areas of the game world and thereby indicate where the player should be looking. 

Another way known to reduce extraneous cognitive load and improve learning outcomes in multimedia 
learning, is cueing the words or events that are important in processing the information, for instance by using 
connectors such as ‘because’ and ‘as a result’; a technique known as signaling (Mautone & Mayer, 2001). Possible 
examples of how to use signaling, cueing and to avoid the split-attention effect in a game context have additionally 



384 




been forwarded by Lawrence (2006). However, as Kriz and Hegarty (2007) noted, telling a student where to look 
does not necessarily imply they comprehend what they see. 

Instructional design guidelines 

In a review of instructional design principles that improve learning with interactive multimedia, Moreno and Mayer 
(2007) state five important guidelines that have shown to increase the efficiency of interactive learning 
environments (but not necessarily games): Guided Activity, Reflection, Feedback, Pacing and Pretraining. Guided 
Activity comprises the use of a pedagogical agent or instructor that encourages the student to actively process, 
organize and integrate new information. The Reflection principle states that students will learn better when 
encouraged to reflect upon the answers provided by the material. Feedback entails that explanatory feedback is 
better than corrective feedback alone, because it provides the student with a proper schema and consequently 
demands less extraneous processing. The Pacing principle states that students will learn better when an animation is 
interactive, in that they can control the pace of the presented content. Pretraining, finally, is meant to activate the 
relevant prior knowledge in students so that they can integrate this with newfound information and subsequently 
create better mental models of the instructional material. 

These design principles are all tried and tested in diverse multimedia learning domains, but the question is 
how they relate to gaming. Of all the principles listed by Mayer and Moreno, the first. Guided Activity, seems most 
applicable to the domain of serious games. In fact, pedagogical agents have already been used in serious games, with 
positive effect. For instance by Conati and Zhao (2004), who used an agent to help students play a puzzle game to 
teach number factorization, or by Moreno, Mayer, Spires and Lester (2001) who had Herman-the-Bug guide 
students through the Design- A-Plant game and thus teach them about ecology. It’s straightforward to see how 
pedagogical agents could work in a game environment that would already incorporate agents or non playable 
characters (NPCs), however getting it right may prove more difficult. Too much guidance and you will take the 
discovery out of discovery learning. Overtly controlling the direction of the game could interfere with the perceived 
autonomy of the player, as well as feelings of presence, which according to Self-Determination Theory harms the 
intrinsic motivation of a game (Ryan, Rigby & Przybylski, 2000). This may not be much of a problem in a step- 
based puzzle game like the two mentioned previously, but more so in a consistent virtual world. It may also come 
across patronizing, or the player will be reminded continuously that it’s an education game, instead of entertainment. 

The reflection principle is applicable to learning in general and states that students will internalize the 
instructional material better if they are encouraged to actively think about the actions they undertook, and the 
associated reasons. As Moreno and Mayer contend themselves however, it is unclear whether interactivity alone is 
already enough to prompt students to reflect on their actions. At least when the objective to be learned is closely 
related to the actions that the player has to perform, they found that, in the Design- A-Plant game, interactivity was 
sufficient incentive for the students to engage in active knowledge construction (Moreno and Mayer; 2005). It is 
plausible that this would not hold if the instructional content is inserted in a game of which the objectives have no or 
only superficial relation with the educational material; for instance, when learning the ecology of plant life on 
another planet was set to the background of shooting aliens, players could finish the game with barely learning the 
material, as was also evidenced by Belanich, Sibley and Orvis (2004). Contrary to Moreno and Mayer’s findings, 
Wong et al. (2007) found no effect of interactivity on learning in their game Metalloman, even though the objectives 
were aligned with the learning content and none of the experimental groups had reflection prompts. A solution to 
this conflicting evidence may lie in what Kiili and Ketamo (2007) call the difference between single-loop and 
double-loop learning. Double-loop learning, contrarily to single-loop learning, involves the formation of a playing 
strategy, and is contingent on the ability to predict the outcome of actions beforehand. 

The feedback principle poses some problems in applicability to games-based learning. Games excel in 
giving corrective feedback on the spot, by means of scores, life-bars, world or NPC behavior, etc. and in fact this 
corrective feedback is to a large extent that what makes games fun in the first place. Giving explanatory feedback 
could be done via a pedagogical agent, but would most of the time require the player to stop his actions, hampering 
the flow of the game, and, like the guidance principle, simultaneously stress that the game is educational instead of 
entertaining. 

User pacing, where the student can control the pace of the educational material, for instance by means of a 
continue button in an animation, will be of little significance to enhancing the effectiveness of games based learning. 
Mostly this is because, due to the inherent interactivity of games, players are already able to pace their progress. In 
so-called cutscenes or textual narrative in the game, where users have little control, it may be important to include a 
continue button. However, nearly all the games on the market already have this on the basis that some people read 
faster than others. 
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The Pretraining principle effectively is what Merrill (2002) refers to as the activation stage, where the 
pretraining activates or provides relevant prior experiences, so that the student can construct meaning by integrating 
the new information efficiently into an existing mental model. With games, the activation stage can be embedded 
into the narrative by having the player ‘glimpse into the future’: in a quiet area a special detail of the workings of a 
complex machine are conspicuously shown in the background, next to a salient object (e.g. a tower), whenever the 
player reaches the machine at a later stage in the game, the tower will activate his or her prior knowledge and could 
help the player understand the workings of the machine and solve a puzzle. Not only is this a way to activate 
relevant domain knowledge and thus improve the effectiveness of the instructional design (Kriz & Hegarty, 2007), 
but mixing the order in which information is presented also fosters affective responses such as curiosity, suspense 
and surprise, which in turn will heighten the enjoyment of the serious game (Hoeken & Van Vliet, 2000). Example 
entertainment games where this has been used to good effect is Nintendo’s Mefroit/ series and Ubisoft’s Prince of 
Persia series, where players get partial hints that help solve later puzzles. 

Promoting knowledge construction by narrative structure 

The guidelines covered in the previous sections roughly pertain to two different approaches to enhance learning in 
multimedia environments. Some of the instructional design techniques mitigate the cognitive overload a student may 
experience during learning, while others have an effect on the overall structure of the instructional narrative, which 
promotes the active integration of the new knowledge into the mental model of the student. The latter pertain to 
what Kintsch calls the predictability and postdictability (Kintsch, 1980) of a text, which are cognitive determinants 
for generating interest and therefore pro-active knowledge integration by the student. As was noted previously, 
being able to predict the outcome of actions is important for the formation of a playing strategy and thereby 
promotes knowledge construction (Kiiri & Ketamo, 2007). High predictability, however, is not always preferable, as 
it will generate little cognitive interest (Kintsch, 1980). In a study by McNamara, Kintsch, Songer and Kintsch 
(1996), a biology text was intentionally made incoherent. While novices to the domain of the paper benefited from a 
highly coherent text, knowledgeable students actually gained a better situational understanding of the incoherent 
text, and scored higher on inference questions, problem-solving and sorting tasks. This shows that, at least for 
people with relevant prior knowledge, moderate unpredictability of the text not only generates curiosity and 
therefore engagement (Loewenstein, 1994), but also fosters knowledge construction, as readers have to actively 
relate and connect different bits of knowledge in their mental model. In a game, moderate predictability could also 
be achieved by changing the chronology of pieces of narrative, by using the example under the pretraining principle, 
or by introducing information gaps. As an example of the latter, imagine a game where you play a fireman and have 
to help a victim with moderate burn wounds while it’s snowing outside. An instructor could tell you to give the 
victim a blanket to stay warm. As an information gap, the game designer could in this case purposely leave out the 
instruction-or blanket, so that the trainee will have to think about a solution himself (e.g. offer his or her coat to 
protect the victim against the cold), thereby internalizing the effects and remedy of cold on a burn victim. 

Postdictability is a term Kintsch uses to designate the ability to a posteriori make sense of the text as a 
whole. If anything strange happened within the text, it should be resolved at the end. As strange events themselves 
create surprise and provoke cognitive interest, the ostensible best way to design the narrative of a game is to cause 
moderate confusion about important elements that are needed for a situational understanding of the instructional 
material, which fall into place at a later stage in the game and are consequently actively integrated into the mental 
model (Hoeken & Van Vliet, 2000). Therefore, one can say that a high postdictability is preferable for serious 
games. 



Towards guidelines for enhancing mental model construction in serious games 

Combining all the points above, we can say that the construction of a mental model can either be obstructed, or 
improved, an overview of which can be found in Figure 1 , as a framework for cognition-based design guidelines 
enhancing mental model construction. In the case of the mental model construction being obstructed, this is 
primarily due to questions of cognitive load. A student can be overloaded with too much information at the same 
time, and therefore miss important information relevant to the integration of new knowledge into his or her mental 
model. In order to mitigate this obstruction, a number of design guidelines can be implemented, roughly pertaining 
to information regulation and the focusing of attention. 

The presentation of information can be regulated, for instance by means of avoiding the already mentioned 
split-attention effect, where some of the information can be offloaded into the other channel. Other means of 
information regulation could be varying the task complexity, or conversely starting out with easy tasks and gradually 
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increasing the difficulty, or imposing or taking away the time limit. Focusing attention can be achieved by 
introducing subtle visual cues, highlighting certain material, or explicit instruction, such as a signpost. It can 
alternatively be done in a game setting by using a pedagogical agent that guides the student on the way, or by 
signaling important events or words in the narration. 

On the side of improving mental model construction, active knowledge construction can be facilitated by 
engineering the predictability and postdictability of the game narrative. Engineering the predictability of the game 
text can for instance be done by introducing information gaps, in order to decrease predictability, and activating 
relevant prior domain knowledge by pretraining in order to increase predictability. Postdictability can be enhanced 
by encouraging students to actively reflect upon actions and by displaying corrective or explanatory feedback. 
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Figure 1 - Framework for cognition-based design guidelines enhancing mental model construction 



Code Red: Triage! 



Until now there is little proof whether the possible design guidelines advanced above transfer to games based 
learning at all. Therefore, we are in the process of developing a serious game with the Half-Life 2 Source SDK, 
called Code Red: Triage, or COgnition-based DEsign Rules Enhancing Decisionmaking TRaining In A Game 
Environment, in order to empirically test some of the design rules hypothesized in the framework above. As the 
context for the game we chose a training of emergency first responder medical personnel, who have to learn how to 
categorize victims according to priority of needed medical attention (a triage), at the site of a terrorist attack on a 
subway station. The reason we chose this type of training is that it incorporates different types of knowledge (e.g. 
verbal, procedural), trains decisionmaking skills while under stress, and elicits affective responses in the student. 
With this we want to test guidelines of each of the four categories in the framework detailed in figure 1, and measure 
the possible effects on learning outcomes and enjoyment of the game. This will then lead to a set of guidelines for 
game designers that may enhance the effectiveness of their serious game. Some work in progress screenshots of the 
game can be seen in Eigures 2 and 3. 
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Figure 2 trainstation area in game 



Figure 3 performing triages on victims 



Discussion 



With this paper we hope to have made the case for finding empirically proven, cognition-based guidelines for the 
(instructional) design of serious games. A lot of research still has to be done however, and some of the findings may 
contradict the hypotheses. For instance, most games now gradually build up the difficulty of the presented 
instructional material, but it may well be that this creates simplified schemata, whereas a larger variability in task 
complexity leads to more active knowledge integration of the student. More generally, a number of the above 
guidelines are intended to only mitigate cognitive overload in the learner. However, there is evidence that it is better 
to optimally load a person, instead of simply decreasing the cognitive load (Wouters, Paas & Merrienboer, 2008). 
This may be even more so for games. While overloading the working memory of a person may inherently be 
undesirable, some people argue that a game is optimally engaging when it completely engrosses and challenges the 
player and the whole of its cognition is concentrated on the task at hand (cf. Sweetser and Wyeth, 2005). As a result, 
a lower cognitive load may result in a less joyous experience; a possible explanation for the existence of unforgiving 
time limits in many games. Within Cognitive Load Theory (Paas et ak, 2003), extraneous cognitive load is always 
seen as something undesirable, but the question is whether this holds for games based learning, where achieving 
flow could be more desirable. Therefore, as extraneous and intrinsic cognitive load are additive, heightening 
extraneous cognitive load when intrinsic load is low and mitigating the extraneous load when intrinsic load levels 
are high, could potentially lead to a more engrossing experience, and therefore a more challenging game. Finding 
such a balance may very well prove impossible however. Furthermore, as was already observed in the learning with 
animations section, the design principles themselves have an influence on the gameplay, can therefore affect the 
overall enjoyment of the game, and should consequently be monitored in future research as well. 

References 



Belanich, J., Sibley, D.E. & Orvis, K.L. (2004). Instructional Characteristics and Motivational Features of a PC- 
based Game. US Army Research Institute for the Behavioral Sciences (Research Report 1822). 

Conati, C. & Zhao, X. (2004). Building and evaluating an intelligent agent to improve the effectiveness of an 

educational game. In Proceedings of the 9* International Conference on Intelligent User Interfaces, 6-13. 

Hegarty, M. (1992). Mental animation: Inferring motion from static diagrams of mechanical systems. Journal of 
Experimental Psychology: Learning, Memory and Cognition, 18(5), 1084-1102. 

Hegarty, M., Kriz, S. & Cate, C. (2003). The Roles of Mental Animations and External Animations in 
Understanding Mechanical Systems. Cognition and Instruction, 21(A), 325-360. 

Hoeken, H. & Van Vliet, M. (2000). Suspense, curiosity and surprise: How discourse structure influences the 
affective and cognitive processing of a story. Poetics, 27(4), 277-286. 

Houtkamp, J., Van der Spek, E.D. & Toet, A. (2007). The influence of lighting on the affective qualities of a virtual 
theater. In Predicting the Future, 25th eCAADe Conference Proceedings, 77-84. Erankfurt am Main. 

Usselsteijn, W.A. (2003) Presence in the Past: what can we learn from Media History? In G. Riva, E. Davide & 
W.A. Usselsteijn (Eds.), Being There: Concepts, effects and measurement of user presence in synthetic 
environments. lOS Press. 



388 





Kintsch, W. (1980). Learning from text, levels of comprehension, or: Why anyone would read a story anyway. 
Poetics, 9(1), 87-98. 

Kiili, K. & Ketamo, H. (2007). Exploring the Learning Mechanism in Educational Games. Journal of Computing 
and Information Technology 75(4), 319-324. 

Kriz, S. & Hegarty, M. (2007). Top-down and bottom-up influences on learning from animations. International 
Journal of Human-Computer Studies, 65(1 1), 911-930. 

Lawrence, C. (2006). Take a Load Off: Cognitive Considerations for Game Design. In Proceedings of the 3rd 

Australasian conference on Interactive entertainment; ACM International Conference Proceeding Series, 
207, 91-95. 

Loewenstein, G. (1994) The psychology of curiosity: A review and reinterpretation. Psychological Bulletin, 116, 75- 
98. 

Mautone, P.D. & Mayer, R.E. (2001). Signaling as a Cognitive Guide in Multimedia Learning. Journal of 
Educational Psychology, 93(2), 377-389. 

Mayer, R.E. & Moreno, R. (2003). Nine Ways to Reduce Cognitive Load in Multimedia Learning. Educational 
Psychologist, 5S(1), 43-52. 

McNamara, D.S., Kintsch, E., Songer, N.B. & Kintsch, W. (1996). Are Good Texts Always Better? Interactions of 
Text Coherence, Background Knowledge, and Levels of Understanding in Learning from Text. Cognition 
and Instruction, 74(1), 1-43. 

Merrill, M.D. (2002). Eirst Principles of Instruction. Educational Technology Research and Development, 50(3), 43- 
59. 

Moreno, R. & Mayer, R.E. (2005). Role of Guidance, Reflection, and Interactivity in an Agent-Based Multimedia 
Journal of Educational Psychology, 97(1), 117-128. 

Moreno, R. & Mayer, R.E. (2007). Interactive Multimodal Learning Environments. Educational Psychology Review, 
79(3), 309-326. 

Moreno, R., Mayer, R.E., Spires, H. & Lester, J. (2001). The case for social agency in computer-based teaching: Do 
students learn more deeply when they interact with animated pedagogical agents? Cognition and 
Instruction, 79(2), 177-214. 

Nelson, B.C. & Erlandson, B.E. (In Press). Managing cognitive load in educational multi-user virtual environments: 
reflection on design practice. In Educational Technology Research and Development. 

Paas, E., Renkl, A. & Sweller, J. (2003). Cognitive Load Theory and Instructional Design: Recent Developments. 
Educational Psychologist, 5S(1), 1-4. 

Prensky, M. (2007). Digital Game-Based Learning. St Paul: Paragon House Edition. 

Ryan, M.R., Rigby, C.S. & Przybylski, A. (2006). The Motivational Pull of Video Games: A Self-Determination 
Theory Approach. Motivation and Emotion, 30(4), 344-360. 

Stanney, K.M. & Kennedy, R.S. (1997). The Psychometrics of Cybersickness. In Communications of the ACM, 
40(8), 66-68. 

Steiner, K.E. & Voruganti, L. (2004). A comparison of guidance cues in desktop virtual environments. Virtual 
Reality, 7, 140-147. 

Sweetser, P. & Wyeth, P. (2005). GameElow: a model for evaluating player enjoyment in games. Computers in 
Entertainment, 5(3). 

Tabbers, H.K., Martens, R.L. & Van Merrienboer, J.J.G. (2004). Multimedia instructions and cognitive load theory: 
Effects of modality and cueing. British Journal of Educational Psychology, 74(1), 71-81. 

Tversky, B., Morrison, J.B. & Betrancourt, M. (2002). Animation: can it facilitate? International Journal of Human- 
Computer Studies, 57(4), 247-262. 



389 




Van der Spek, E.D. & Houtkamp, J.M. (2008). Mixing Emotions, How Physical Discomfort Influences the Affective 
Appraisal of Virtual Places. In 2008 International Conference Visualisation, 18-22. 

Van Merrienboer, J.J.G. & Sweller, J. (2005). Cognitive Load Theory and Complex Learning: Recent Developments 
and Euture Directions. Educational Psychology Review, 77(2), 147-177. 

Van Oostendorp, H., Beijersbergen, M.J. & Solaimani, S. (2008). Conditions for learning from animation. In 
Proceedings of the International Conference of the Learning Sciences (ICES). Utrecht: Utrecht 

University. 

Vogel, J.J., Vogel, D.S., Cannon-Bowers, C.A., Muse, K. & Wright, M. (2006). Computer Gaming and Interactive 
Simulations for Learning: A Meta- Analysis. Journal of Educational Computing Research, 34(3), 229-243. 

Wilson, K.A., Bedwell, W.L., Lazzara, E.H., Salas, E., Burke, C.S., Estock, J.L., Orvis, K.L. & Conkey, C. (2008). 
Relationships Between Game Attributes and Learning Outcomes. Simulation and Gaming OnlineFirst, 
published August 26, 2008. 

Wong, W.L., Shen, C., Nocera, L., Carriazo, E., Tang, E., Bugga, S., Narayanan, H., Wang, H. & Ritterfeld, U. 

(2007). Serious Video Game Effectiveness. In Proceedings of the international Conference on Advances in 
Computer Entertainment Technology (Salzburg, Austria, June 13 - 15, 2007). ACE '07, vol. 203. ACM, 
New York, NY, 49-55. 

Wouters, P., Paas, E. & Van Merrienboer, J.J.G. (2008). How to Optimize Learning Erom Animated Models: A 

Review of Guidelines Based on Cognitive Load. Review of Educational Research, OnlineFirst, published 
August 27, 2008. 

Wouters, P., Van der Spek, E.D. & Van Oostendorp, H. (2008). Current Practices in Serious Game Research: A 
Review Prom a Learning Outcomes Perspective. In T.M. Connolly, M. Stansfield & L. Boyle (Eds.), 
Games-Based Learning Advancements for Multisensory Human Computer Interfaces: Techniques and 
Effective Practices. Hershey, PA: IGI Global. 



390 




A Case Study in Japan about Effects and Issues of Instructional Media Center 

in the University 

Morio Yoshie, Ph.D. 

School Education, Graduate School of Comprehensive Human Sciences 
University of Tsukuba, Japan 

At present, it is very important to introduce instructional technology to university teaching. In the present 
report, the author merged the findings of his 14 reports written in period 1986-2008 which is closely related to the 
work done in the section of a university he belonged, and discussed the effects and issues of instructional technology 
in the university. 



1. Background of the study 

In the Japanese university the author had been worked in, the ‘Educational Media Center’ (EMC) was 
constituted during 1975-2003. The center was consolidated into the center ‘Academic Computing and 
Communications Center’ (ACCC) which obliged to manage information communications technology and media 
systems in 2004. In the present report, the author merged the findings of his 14 reports concerning his work in EMC 
during 1975-2003 and in ACCC thereafter, and tried to discuss the effects and issues of instructional media center in 
the University. 

The objectives of EMC were to manage educational equipments in the university, advice for effective use of 
media, develop educational media, produce educational materials, thereby contribute for improvement of media use 
in the university. In the present report the author describes, not the entire work of EMC, but the findings he gained 
during his employment to EMC in working as one of the team of projects. His carrier consists of graduate course of 
experimental physics, working in EMC and currently majors the educational technology in the Institute of Education 
of University of Tsukuba. 



2. Analyses of reports during 1975-2008 

(2.1) Development of the instrument for the physics experiment *'*®*^* 

In 1987, the Borda’s pendulum was a theme of college physics experiment in the University of Tsukuba. Digital 
timers that work as the peripheral unit of microcomputer were developed. Connecting them to microcomputer and 
sensor, apparatus for measuring pendulum period were constructed. We used them in the student experiment. We 
could also analyze the mechanism of fluctuation in the period and effect of string mass in calculating g value using 
the apparatus. The development of learning materials grew into physics education study. 

(2-2) Development of the timer unit as the peripheral device of microcomputer 

Three digital timer units, for use when microcomputers are being used in secondary school and college 
science experiments, have been constructed on a trial basis. These timers work as peripheral units of 
microcomputers, have counting capacities of 8 digits or more, and hold counter values when commands from 
microcomputers are given or when external signals are supplied. The values held may be read into the 
microcomputers. In this sequence of operations, the timers continue counting. The counter IC’s (integrated 
circuits)used are: 8253 (product of Intel), MSM5512RS (Oki), and ICM7227 (Intersil). Empirical comparisons are 
made between three timers based on composition of each experimental circuit, interface to microcomputer, control 
program, functional difference, ease of construction and cost. 

(2-3) Development of the data processing software for language test score 

Data processing system as a part of English language testing was developed. The author had joined a project 
about language testing and made several computer programs to manage data of testing. Although he could not 
naturally be the subject of the study, those programs worked effectively for several years in processing the data of 
testing. 
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(2.4) Installation of local area network for human science major 

In 1989, at the beginning period of introduction of Internet to Japanese University, we started to install the local 
area network (LAN) into the human science section. The author described the record of initial stage installation of 
LAN for human science major. The university computing section managed the central part of LAN and each major 
in the university managed the peripheral part of the LAN corresponding to its section. The hardware of LAN 
constituted of Ethernet yellow cables and the repeaters which were connected to the router managed by the 
computing section. In 1989, we introduced the SPARC Station 1+ workstation of Sun-microsystems. Inl994, the 
SPARC Station 5 was introduced. Although faculty members of human science were the novice of computer science, 
they worked to install the network in order to establish good research environment. In this instance, the user of 
comuting sention had to manage the information infrastructure partly by themselves. 

(2.5) Use of the audio-visual media in the physics experiment 

The author referred the concrete example of Barton’s Pendulum of Nuffield Physics and the use of picture for 
the observation of the experiments. Students learned forced oscillation using the apparatus similar to those referred, 
the video through computer network, and the graphics generated by theoretical model. The author discussed about 
the effects and restrictions for student’s experience expansion by using the Audio-visual media. 

(2.6) Relation between information processing of specific major and Informatics education *** 

Since 1992, the author had been covered a part of the college class about informatics education for teacher 
preparation curriculum. He asked students in the class to write report on what he/she know about information 
processing or information technology in her/his major. The author also asked students to write how he/she could 
utilize those knowledge for the future activity of teaching. Three examples of information processing were 
discussed; information processing for archaeology, the corpus for applied linguistics, and the geographical 
information system. The author discussed the advantage of adopting those materials in high school subject 
‘Informatics Education’ . 

(2.7) Use of network in distance education for research guidance in school of education 

An experiment during 1999-2001 that was conducted to gain an insight into the state of distance education in 
the doctoral program of a school of education: Research guidance in the experiments was conducted in a seminar- 
type format using distance communications network. After the presenters (graduate students) had made their 
research presentations, the students, tutors, and teachers would discuss them, and research guidance was given 
during that time. In order to provide research guidance under these conditions, it was determined that the network 
requires such functions and features as confidentiality, moving images where expressions are easily understandable, 
clear audio, no problem with signal transmission or overlap of delayed audio signals, sharing of thought-provoking 
images, and the ability of instructors to switch between Internet lines and screens. We should emphasize that 
universities will have to undertake distance education as projects. 

(2.8) Educational use of science video 

Education in science and technology is currently an important social issue. A set of video programs about low 
temperature physics produced by university teaching staff was shown to elementary, junior high, high school and 
university students and teachers, who were asked to reply to a questionnaire. Using this video about a particular area 
of science, viewpoints and opinions on science videos were collected. These were then classified into the following 
categories: effects on the cognitive domain, position in educational activities, possibility of use as teaching 
materials, effect on career choices, and opinions on video production. We consider that these concrete viewpoints 
and opinions gained from one particular area of science, low temperature physics, will give hints for the use of 
videos in practice. 

(2.9) Necessary Conditions for Effective Use of e-Learning in the University 

In the present investigation, the author tried to search the necessary conditions for effective use of e-Learning in 
the University especially in the University in which he belongs. The methods of study are to analyze several recent 
literatures and to describe the experiences he gained during his educational activities in which he utilized 
educational media. In the present report, the author hypothetically listed the objectives to utilize e-Leaming as : 1) 
To upgrade the educational functions of undergraduate and graduate level of the University, 2) To appeal the 
University to the society by developping the open courseware while utilize the OCW to the University education, 
and 3) To increase the learning oppotunities for students from abroad. 
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As the results, following items are obtained as the clues towards the research objective: 1) The examinatuion of 
teaching / learning resources and educational methods in the University, 2) How should be the role of the divisions 
obligated to install and maintain the information communination technology and audio-visual media, 3) Installation 
of course manegement system and systems for producing e-Learnig contents, 4) Framework of support to produce 
the e-Learning materials in the University, 5) To consider wide aqpplication of e-Learning, 6) Direct contribution of 
technical staffs to the reserach / educational activities, 7) Nurturing experts of e-Learning, and 8) The examination 
of contents and methods of training courses conducted in the company. Studies of those items could help the 
effective use of e-Learning in the University. 

Following research subjects would be very important for educational researcher, that is, applying e-Learning 
method properly to education field in obligation, proposing requirements and functions of e-Learning system which 
are crucial for education, and identifying functions that e-Learning could play in the whole method of teaching in the 
University. 

(2.10) Necessary Items for Renewal and Operation of Educational Media in Large University Lecture Rooms 

( 12 ) 

An analysis of necessary items for renewal and operation of educational media in relatively big general 
university lecture rooms: According to hopes and needs by teachers and students collected, it is necessary for 
departments in charge to collaborate in installation and renewal of educational media system based on the long term 
planning. That is, the management, educational media department, school affairs division, facilities division should 
collaborate to produce good system. Financial executive of the university should distribute reasonable budget and 
personnel for educational media. 

Around the time and after two centers are unified into one, the service for educational media for large lecture 
rooms were stopped. It was considered by the managements in the university that installation and maintenance of 
educational equipments to those rooms are obliged to each graduate school. According to the observation of the 
author, it is necessary that the media specialists support those tasks. 

3. Concluding remarks *^^**^‘** 

The author analyzed the conditions for effective use of media in the University and the role of the center. The 
educational equipments used were the equipments in the classroom, audiovisual equipments, apparatus for physics 
experiment, hypermedia, database, LAN in human science major, distance communications network, and so on, 
based on information technologies and media technologies. The educational fields were the University lectures, the 
Japanese learning materials for students from abroad, the evaluation method and English learning materials for 
Japanese students, the teacher education, the distance education for research guidance in doctoral programs of 
school of education, informatics education and so on. Five instances follow. 

The author has been worked for several developmental projects for educational use of computer as on of the 
project member in a specific university. He tried to describe those projects from multiple view points: structure of 
computer system, production of instructional material, supplementary electric work and so on. Five projects 
practiced during 1976 to 2004 were selected as the instances. Following results were induced, (a) Concerning the 
production of teaching materials, educational technologist can not touch the content of them, if his/her major were 
not the subject of interest. In spite of this situation, educational technologist must promote production of teaching 
materials, (b) Also it is necessary to think ahead the period that the teaching materials are effective, the copyright 
transaction, possibility of publishing and so on. (c) It is important to make the method of collaboration between 
instructors, contents specialists, production specialists, those who in charge to install computer systems. It is 
necessary to create the collaborative way to share the role in which each person has her/his specific results or fruits. 
If we refer the discipline, collaboration between the information science/technology, the contents majors, education 
major are indispensable. 

The review suggested that the role of instructional media center includes to build the ways to coordinate 
activities to assure autonomous work of each participant in the joint project in which content specialists and media 
specialists work together, to positively motivate media specialist who has no direct concern with teaching activities, 
and to practice effective systematic services by media specialists. Especially it is necessary to establish conditions 
for active participation of contents specialists. In the context of this study, the value of the product of educational 
technology could be found in the practice and the investigation of effective use of educational media system as a 
part of educational method. 
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Major of the author in his graduate school was experimental physics. In employment in the educational media 
center, he engaged many projects as one of the teams. He remembers that productive work were done when the 
content of the project was connected to his old major. His old major is merely one of disciplines in the university. 
And each discipline has needs to embrace aspects of media use. So he conclude that instructional media center 
should accept collaborative work with each major of the university and make good conditions for such utilization of 
the center. 
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Introduction 

Current educational movement towards integration of (multi)media technologies into teaching and learning 
together with vast its wide availability in the market call into the needs for educators to critically examine the 
technologies being used beyond the examination of matters such as quality of graphics, factual accuracy of the 
content and the amount of Mathematical skills required (Posner, 2004). In addition to the strategy of determining the 
relative needs and advantages that technologies could accomplish to support teaching and learning (Roblyer, 1997), 
educators need to critically examine the aspects of curriculum that underlies the production of an instructional 
(multi)media in order to help them to justify its usage in their unique teaching and learning contexts. 

This paper presents the curriculum analysis of CyberChase, a popular Mathematics-driven television 
program targeted to children age eight to twelve years old, and/or third to fifth grade funded by the National Science 
Foundation (NSF) using Posner’s (2004) framework of “The process of curriculum analysis”. Expanding on Tyler’s 
Rationales for Curriculum Planning (Tyler, 1949) as the main framework, Posner suggested four sets of components 
or “questions” pertaining to analyzing a curriculum; (1) Curriculum documentation and origins, (2) Curriculum 
proper, (3) Curriculum in use, and (4) Curriculum critique. This paper addresses each of the components and relates 
it to the design principles undertaken by CyberChase production team. 

CyberChase: Learning through media and technology 

CyberChase is an Emmy award winning (Donlevy, 2004; Olsen, 2006), Mathematics-driven animated 
television series designed for children ages eight to twelve years old funded by the National Science Eoundation 
(NSF) under the “Informal Science Education” category, among other major funding agencies (CyberChase, 2008a). 

The CyberChase series are produced by the Thirteen/WNET New York and Nelvana International, and 
broadcasted daily on 340 Public Broadcasting System (PBS) stations since 2001 (CyberChase, 2008a). In Iowa, it is 
broadcasted via the Iowa Public Television (IPTV) at Channel 11. At the time this paper was written, CyberChase is 
now entering its sixth season with the campaign, “Math in Action”, an attempt to link Mathematics with Science 
learning experiences (Thirteen, 2007). 

The plot of the series is simple. In each episode, the children (i.e. learners) will be presented with a problem 
caused by a villain named Hacker and his two assistants. Buzz and Delete. The three heroes named Inez, Jackie and 
Matt who are about the same age as the target audience and their cyber bird helper, Digit, attempt to solve the 
problem and save the Motherboard planet. Flowever, they need to apply Mathematical concepts and skills in order to 
save the planet since Hacker is very good at capitalizing on his Mathematical knowledge and skills to cause the 
planet’s destructions. 

Each episode of CyberChase is followed by a special segment called For Real. In this five-minute segment, 
children and/or audiences are presented with the real-world application of the Mathematical knowledge and skills 
presented (i.e. “taught”) in the episode. The children will be presented with a scenario or dilemma experienced 
Bianca and Harry, stars of the segment, in their everyday lives that requires the application of the Mathematics 
concepts highlighted in the episode. This segment is one of the important components of the television series as it 
assists in scaffolding children to view Mathematics knowledge and skills as practical and contextual-based concepts. 

Theoretical perspective 

As a multimedia project that utilizes technology to teach Mathematics as content knowledge, CyberChase, 
in the form of a television program is viewed as an educational tool that shapes the informal learning experiences, 
which is “learning that occurs outside the classroom, unofficial, unscheduled, and often occurs in impromptu way” 
(Cross, 2006). This study views the CyberChase television program beyond the conventional education tool; it is 
viewed as an “extra curriculum” that comprises all of those planned experiences outside of the school subjects which 
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are responsive to students’ interests and while not hidden, openly acknowledged dimension of the school experience 
(Posner, 2004). The term “experiences” in this paper refers to (Schwab, 1978) view of the five bodies of learning 
experiences that needs to be presented in a process of curriculum revision; teacher, learner, milieus, subject matter 
and curriculum making. Table 1 below presents the adaptation of Schwab’s (1978) five bodies of learning 
experiences with its representations in CyberChase. 



Schwab’s five bodies 
of learning 
experiences 


Descriptions 


Representation of Schwab’s five bodies 
of learning experiences in CyberChase 


1. Teacher 


Knowledge of the teachers, including 
teachers’ knowledge, characteristics, 
personalities, readiness to learn new 
materials and new ways of teaching 
(Schwab, 1978b, p.367) 


The Big Idea or learning objective in 
CyberChase which is linked to the National 
Council of Teachers of Mathematics 
(NCTM) Standards represents the ‘teacher’ 
because it defines the objectives to be 
achieved and the ways of how it can be 
achieved, the tasks that usually carries out 
by teachers. 


2. Learner 


Knowledge about the learners, divided 
into two: (1) knowledge about the 
learners’ present state of mind and heart, 
including the age group, prior knowledge, 
readiness to learn, aspirations and 
anxiety, and (2) knowledge about the 
learners’ development towards being an 
adult (Schwab, 1978b, p.366) 


The viewers of CyberChase television 
program are the learners. They can include 
the children, their parents and family 
members, teachers and everyone who 
watches the television program. 


3. Milieus 


The milieus of which the child’s learning 
will take place and “in which its fruits 
will be brought to bear” including the 
school and classroom of which learning is 
supposed to occur, the child’s interaction 
with the family, community, and 
particular groupings of religious, class or 
ethnic genus (Schwab, 1978b, p.367). 


The milieus is represented by the 
technology, specifically the 30 minutes 
television show, broadcasted daily. 


4. Subject matter 


Knowledge about the content and area of 
learning (Schwab, 1978b, p.367) 


The subject matter covered in the 
CyberChase series are Mathematics, 
Science and Technology Education 


5. Curriculum making 


Knowledge on managing the process of 
curriculum making among the four bodies 
of representation of learning experiences: 
teacher, learner, milieus and subject 
matter (Schwab, 1978b, p.367). 


Members of the CyberChase production 
team are actively engaged in designing, 
creating, testing and involve in every aspect 
of curriculum making decisions 



Table 1: Schwab’s (Schwab, 1978b) five bodies of learning experiences and its representations in the CyberChase 
television program 



The analysis of the television program was guided by Posner’s (Posner, 2004) “Process of Curriculum 
Analysis”. Expanding from Tyler’s (1949) “Rationales for Curriculum Planning”, Posner suggested four sets of 
curriculum analysis: (1) Curriculum documentation and origins, (2) Curriculum proper, (3) Curriculum in use, and 
(4) Curriculum critique. Each of these components contains a set of questions that need to be answered. The set of 
questions relevant to this study is represented in Table 2 below. 
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Component(s) 


Questions 


Curriculum documentation and 


How is the curriculum documented? 


origins 


What situation resulted in the development of the curriculum? 
What perspectives does the curriculum represent? 


Curriculum proper 


What are the purposes and content of the curriculum? 
How is the curriculum organized? 


Curriculum in use 


How should the curriculum be implemented? 

What can be learned from an evaluation of the curriculum? 


Curriculum critique 


What are the curriculum’s strengths and limitations? 



Table 2: Posner’s (2004) framework on curriculum analysis 



Methods of analysis 

Working from qualitative research mode of inquiry, the main methods used in this study were media and 
document analyses. The researcher watched and observed selected episodes from the CyberChase series based on 
the specific themes that the series addressed. For each episode, the researcher took notes on its structure, timeline, 
instructional objectives and storyline. 

In addition, the researcher collected and analyzed information on the television series that are available on 
the series’ website (http://pbskids.org/cyberchase), including the information under the “Parents and Teachers” 
section that explained the objectives of each of the episode and suggested ideas on how parents and teachers could 
use them to expand Mathematics-based discussions and learning with their children at home and schools. Other 
supplemental information includes the advertisements related to CyberChase series in journals and publications, and 
information from the National Science Foundation (NSF) website regarding the purpose, history and any other 
relevant information allied with the series’ production. 

Findings and interpretations 

Media and content analyses of CyberChase revealed that this television series has the potential to become 
an educational multimedia tool for Mathematics learning especially for the elementary school level. The major 
contribution of this series is its ability to connect Mathematics to everyday living experiences which allows children 
to see the real-world application of Mathematical concepts and principles beyond classroom walls. This section 
presents some of the major findings of the study. 

Component 1: Curriculum documentation and origins 

Analysis of the television series revealed that each of CyberChase episodes is aligned with the National 
Council of Teachers of Mathematics (NCTM) Principles and Standards for School Mathematics Grades 3 to 5 
(NCTM, 2004). Each CyberChase episode features one Big Idea which is linked to one of the NCTM’s Expectations 
in a specific Math content area. For example, in the episode 203 Harry Hippo and the Mean Green, the Big Idea is 
“Fractions that look different can represent the same portion of a whole” (CyberChase, 2008a). This Big Idea is 
linked to NCTM’s “Numbers and Operations” content area. 

The educational philosophy of “Math is everywhere and everybody can be good at it” became the major 
foci in the design and development of the series. Consequently, it infused the hidden agenda, or hidden curricula, 
that the children have the autonomous power over their learning and the way they solve problems. It also attempts to 
instill positive attitudes to the children such as ‘never gives up’ . For instance, in episode 202 Totally Rad, that 
focuses on the concept of perimeter and area relationship (CyberChase, 2006), one of the animated characters. Matt, 
had almost gave up when he learned that Hacker had changed the perimeter of the area/field of the skate-off 
competition. His friends, Jackie and Inez, supported Matt to continue with the competition. Inez said, “That doesn’t 
mean you have to give up. Matt. Team Motherboard never gives up!” (CyberChase, 2008a). 

Component 2: Curriculum proper 

Since the Mathematics content in CyberChase is delivered in entertaining and fun ways, there might be 
possibilities that the children will not realize the Mathematical concepts that they need to learn from each episode. 
As a solution to this concern, the production team created a 10 minute segment called For Real to demonstrate how 
Mathematical concepts featured in each episode works in real-life situation. 

For example, to further enhance children’s understanding of the concept of fractions in the episode 203 
Harry the Hippo and the Mean Green, the children were presented with a For Real segment called Bianca ’s New 
Pet where Bianca, one of the teen characters in the series, need to figure out how different pieces and sizes of woods 
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can make up a whole roof for her doghouse. This situation strengthens children’s prior learning on equivalent 
fractions featured in the 203 Harry the Hippo and the Mean Green episode. 

Component 3: Curriculum in use 

This study concluded that CyberChase television series are capable of being a tool to expand Mathematics 
learning beyond classroom especially with the creation of CyberChase website. The CyberChase website provides 
basic information about the television program and its characters. Mathematics-related educational games, materials 
and resources that are connected to the series episodes (Donlevy, 2004). 

Of particular interest a special section designed for parents and teachers that include practical suggestions 
and instructional materials on how to reinforce students’ learning. The ready-made lessons and activities in this 
section help teachers and parents to incorporate Mathematics concepts in and/or outside classroom and at home. 

In addition, the Tips and Support section in the website provide the opportunity for parents and teachers to 
communicate with the “experts” of Mathematics learning. There is also a section in the website where the parents 
and teachers can help the children to throw a CyberChase party using CyberChase materials, for instance, the theme 
song, the Math trivia etc. (CyberChase, 2006). All of these tools enabled CyberChase to be considered as a 
community engagement tool. 

Component 4: Curriculum critique 

Despite of the many positive elements associated with the CyberChase television series including the 
positive results of the impact of the series on children’s Mathematics learning (CyberChase, 2008b), there are also 
some concerns about its impact on majority of children. Since the series are aired via Public Broadcasting System 
(PBS) stations, it reaches only to limited number of viewers compared to other commercial network stations such as 
Cartoon Network or Disney Channel. 

Another concern is the underutilize application of this television series as a learning tool integrated within 
current curriculum in schools. Analysis of various research reports conducted by a group of independent researchers, 
for instance, (Flagg, 2003a, 2003b) as well as analysis of multimedia resources on the use of CyberChase series 
among teachers revealed that application of these series is limited to out-of-school programs. These results indirectly 
imply that CyberChase series function as supplemental instructional aids in Mathematics teaching in spite of its 
valuable features and potentials to be one of the most effective learning tools for children. Perhaps one of the 
reasons is due to the lack of exposure among educators and parents about what the series is all about and how it is 
designed to meet National Council of Teachers of Mathematics (NCTM) standards and principles. Another reason 
could be due to limited access to technologies such as television and Internet in certain schools, especially among 
socially and economically disadvantaged neighborhoods throughout United States. 

Conclusion 

In one aspect, this study attempts to introduce multimedia technology as another component to be 
considered in Posner’s (2004) seven common concepts of curriculum based on the arguments presented earlier in 
this paper. In addition, this study also reveals the potential of CyberChase television series to be an effective 
Mathematics learning tool along with other learning tools by decomposing the principles that guidance each design 
and development of the series. 

It is hoped that by unmasking the design principles behind its production, more educators and parents will 
discover the advantages of using CyberChase television series for Mathematics instruction and to be able to 
integrate it into current in-school (not just after-school) teaching and learning settings along with the official and 
operational curriculum they implemented in the classroom. 
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Introduction 

The importance of using models of systematic instructional design in education has long been recognized 
for the purposes of enhancing learning and instruction. A systematic instructional design approach improves the 
management of instructional design and development processes, the evaluation and testing, and builds on associated 
learning or instructional theories (Andrews & Goodson, 1980). However, instructional designers and educators 
always struggle as to which model to be used as a guideline when designing and developing instructional 
(multi)media for real clients and audiences. Model understood, here, as a mental picture that helps us understand 
something we cannot see or experience directly. Since instructional design is a dynamic, needs- and contexts-based 
process, there is an obvious lack of detailed, practice-based literature on instructional design processes and 
procedures. At the same time, many challenges that instructional designers go through when designing instruction 
and completing their work are not extensively documented. 

This paper is a small effort to fill in the gap. Aligned with the intention to shine the rays of change in the 
practice of instructional design culture, this paper proposes a practice-based instructional design (ID) model built on 
the experiences of working as instructional designers with a group of clients and subject matter experts in a teacher 
training program. Instructional design concerns, issues and ethics are also discussed in detail. 

Methodology 



Setting 

This study took place in a large research university in the Midwestern United States with a recognized leading 
program of instructional technology in teacher education (Davis, 2003). This same university offers Masters and 
PhD programs in Curriculum and Instructional Technology designed to meet the needs of professionals in education 
seeking leadership positions in instructional technology. 

Students in this program are required to enroll in a 3-credit hour graduate course in instructional design. 
This course introduces students to theories and models of instructional design. Part of the assessment of this course 
is an instructional design project where teams of students are required to create instructional experiences for 
identified community members. Students need to specify a real-world client for their projects, choose an 
instructional topic to be developed, conduct a needs, learner and context analyses, develop a design blue print, 
create instructional materials, deliver the instruction and conduct a formative evaluation (Correia, 2008). Since the 
nature of the project relies closely with clients’ commitment and consent, students are challenged to work with 
clients all the time in order to complete the project. 
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The project 



The case described here took place in the above setting during fall 2006. A mixed group of four beginner 
and intermediate-level instructional designers (i.e. students in the course) were involved in designing and developing 
a tutorial-based instructional material for pre- and in-service teachers (i.e. target audiences) on integrating KidPix 
software into reading and language arts curriculum. The client of this project was a faculty member who taught an 
undergraduate course on integrating technology into reading and language arts curriculum. In the process of 
designing and the developing the solution, the design team worked with another faculty member in the department 
who acted as an expert in teaching literacy and literacy teaching methods.. This faculty member served as the 
Subject Matter Expert (SME) for the team’s project . Additionally, the course instructor served as the SME on the 
instructional design process. 

The instructional designers’ design emerged from the complex relationships between addressing the needs 
of the clients and the pre- and in-service teachers, dealing with the SMEs, and the demands of a rigorous and 
systematic design process. These relationships are explored in this practical instructional design model. 

As for the design and production of the instructional solution, all team members were directly involved as 
participants-observers in the processes of designing, developing, implementing and evaluating the instructional 
materials. Eour types of analyses were conducted: content analysis, goal analysis, learner analysis and needs 
assessment, and context analysis in the forms of post-test surveys and interviews with the ID experts, clients, target 
audience and Subject Matter Experts were used to gain information on learners’ mastery of skills. 

Mode of inquiry 

Based on Schon’s (1983) perspective of practice-as-inquiry, this study applied narrative inquiry as its 
research strategy to give “new and deeper insights into the complexity of practice contexts’’ (Riley & Hawe, 
2005).Authors of this paper engaged in the process of reconstructing the incidents as suggested by Clandinin & 
Connelly (2000) and reflected on their experiences working with students, clients, real instructional needs and 
different SMEs as a designer in training and course instructor . Specifically, the designer in training reflected on the 
conflicts and challenges she encountered in the process of negotiating and balancing between client’s needs and 
SMEs’ recommendations while the course instructor reflected on her role as course designer/developer and team 
facilitator who provided practical suggestions to resolve the conflicts and challenges encountered along the way. All 
of these reflections contributed into the development of the proposed ID framework. 

In addition, an analysis of the team’s project reports was conducted. Special attention was given to the 
final project report, which detailed the instructional design plan and decisions undertaken in the instructional design 
process as well as a justification of the modifications made to Braden's (1996) instructional design model that was 
applied in the beginning of this project, but which evolved into a different model by the end of it. 

The development of the proposed ID model 

The proposed ID model (Diagram 1) presented the detailed information regarding the processes and outputs 
involved using the basic ADDIE (Analysis, Design, Development, Implementation and Evaluation) cycle as the 
guideline. It was initially adapted and later modified from Braden’s (1996) instructional design model. 

Modifications were made to the processes involved based on the designers’ experiences working with SMEs and the 
results of usability testing and formative evaluation conducted. In addition, this ID model proposes the combination 
of both the Design and Development phases because from this project experiences, both phases were inter-related 
and sometimes iterative in nature. The most important element that distinguishes this model from Braden’s (1996) 
model is that each final design decision was subject to the instructional designer team members’ approval with 
consideration of selective SMEs’ feedback. 

Analysis phase 

In the analysis phase, need analysis was conducted to determine the need for instruction. If a need existed, the 
design team should clearly identify the need, conduct a goal, learner and context analyses, which would result in 
production of a statement of instructional needs and goals. A variety of data collection methods (e.g., interviews, 
questionnaires and observations) should be used to gather information representative of all different dimensions of 
analysis. The instructional design process ends if no instructional need(s) was identified. 

Eormative evaluation (E.E) 
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Formative evaluation was carried out in the analysis phase specifically during the process of writing the instructional 
needs and goals, and mastery knowledge and skills. An important element in this process was the active involvement 
of the SME and client(s), who would provide their expert opinions and feedback on the identified instructional 
goals, knowledge and skills to be mastered. However these opinions and feedback were subject to the approval of 
the design team. 

Design and development phase 

The design and development of instructional strategies were always consistent with the defined instructional 
analysis. Instructional designers worked closely with the SMEs in deciding which instructional strategies to be 
applied and in producing the blueprint of the instructional strategies, which served as the main input for the 
prototype creation. The finalized prototype would then be pilot-tested with a group of subjects who have 
characteristics quite similar to the actual target audiences (i.e. learners). Participants in the test would be asked to 
complete a functional analysis checklist of the prototype as a way to determine the effectiveness of the prototype. 
The final output of this phase was the finalized instructional materials designed and developed based on the results 
of the instructional analysis process. 

Implementation phase 

Implementation of the instructional would be carried out with the actual target audiences (i.e. learners). This process 
may involve production of training manuals for the instructor (i.e. trainer or teacher) if needed containing detail 
explanation on how instruction should be carried out. 

Evaluation phase 

Ideally, at the end of instruction, learners were expected to complete a summative evaluation (or post-project 
evaluation) on the topic addressed (remind the reader which topic it was). The summative evaluation tools might 
include a short survey, interview with in- and pre-service teachers and any other methods, like class observations. 
Learners’ feedback would be the input in the generation of the final report, which included recommendations for 
further development. 
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Diagram 1 : The proposed ID model 
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Interpretations and Conclusions 



The main feature the proposed ID model is the collaborative working relationship between the SMEs and 
the designers in the design and development phases. The instructional designers would first consult with the clients 
to identify their needs for instruction, and engaged the SMEs in the design proposal review, sought for their 
comments, approval and suggestions, and finally made appropriate changes before proceeding to the next design 
phase. This iterative process is the uniqueness of the model as it represented the mode of working with the experts as 
participants in the design and development process whereas traditionally, experts usually functioned merely as 
content providers without much participation required from them as part of the instructional design team. Even 
though SME is considered as part of the design team in this model, their involvement is limited only in the analysis, 
design and development phases and their involvement and feedback in the project would always be negotiated and 
balanced with the designers’ instructional decision based on the client’s needs. 

Another important essence of the proposed ID model is the production of written reports as outputs in the 
analysis and design/development phases. By documenting the outputs in a written form, the design team would be 
able to refer and reflect on its content before proceeding to the next phase. In addition, these documents assisted in 
keeping a steady flow of documentation on design and development processes as well as maintaining the client 
informed and involved at all times. 



Practical Implications 

Eollowing are some of the main lessons learned when following this practical ID approach as a means to 
improve instructional design culture and practices: 

1 . active participation from SMEs as part of the instructional design team contributes to successful 
production of instructional multimedia; 

2. while SMEs has an important voice in the design and production of the instructional multimedia; their 
opinions are still subject to the decision of the design team based upon the needs and interests of the 
major client for the project; 

3. carefully designed formative evaluation (e.g., usability testing) is needed in order to gain maximum 
input and feedback from the potential learners; and 

4. it is essential to use a prototype that truly reflects the real intended instructional multimedia product 
when conducting formative evaluation. 
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Effective Application of Computer Game Technology in K-12 



Yadi Zieehezarjeribi 
Paige Worrell 
Ingrid Graves 

Abstract: This study explores the use of computer games and simulations in K-12 
environments and provides a basic rationale for the use of video games to improve problem-based 
learning. With the physical infrastructure of computer technology in place, simulations and games, 
become a viable source for engagement of student learning through repetition and practice. Using 
experiential play methodology with 14 middle school students as they engage in playing Medal of 
Honor and Call of Duty, (first-person shooter video games). We identified key characteristics of 
the game that correspond to engaged academic learning using the 5E Learning Cycle (Kolb, 1984) . 
Implications for the use of computer games and simulations in classrooms are examined. Initial 
data analysis demonstrates that students are more willing to make mistakes in a game than in the 
classroom. Avatars provide a risk free setting through which students may engage in problem 
based learning. This is held in direct contrast to students who disengage from learning when they 
feel unsuccessful within traditional instructional settings. 

Introduction 

According to Barab, Warren and Ingram-Goble (2006), many schools are looking for research to determine 
if video-gaming has a direct impact on student learning. While notions of the efficacy found in the play of some 
games is informative (Appelman & Wilson, 2005), seldom does having fun playing games motivate a principal to 
incorporate a game into the curriculum. Our focus is how to effectively use video games to support classroom 
instruction. 

Using Computer Games to Enhance Teaching and Learning in K-12 

Vogel, et.al. (2006) note that methodologically sound research into the effects of gaming and simulations in 
education has been extremely limited. There are several studies that have shown to support academic achievement. 
Eor instance, Barab et. al. (2007) found that through the use of Quest Atlantis, a 3D virtual world, “students showed 
statistically significant gains, demonstrated rich insights in terms of their submitted work, were clearly engaged, and 
participated in rich scientific discourse. Eurthermore, through participation in this narrative students developed a 
rich perceptual, conceptual, and ethical understanding of science” (p.76). 

Rosas, et. al. (2003) demonstrated significance in first and second grade scores. This study showed an 
unexpected variable of increased attendance. As a result, the researchers had difficulty deciding whether the increase 
in school performance was due to the use of GameBoy or the fact that the GameBoy was used as an extrinsic 
motivation for increased attendance. 

Thiagarajan (2003) suggests that “many people are desperately seeking research evidence to prove that 
training games are more effective than traditional strategies” (p.2). Results found by many of these studies have 
been inconclusive; perhaps the answer to engaging inquiry must begin by first examining what works and 
capitalizing on the success of a few projects. This study provides a preliminary report from an ongoing study of 
educational gaming using the following research question: How can video games be used to supplement classroom 
learning activities? 

Because we are looking at the learner’s subjective experience during game play, the 5E's Learning Cycle 
(Bybee, 2002) was included as part of the framework for this study. It is a model for instructional design based on 
the experiential learning cycle (Kolb, 1984). There are five parts in the original model (Engage, Explore, Explain, 
Elaborate, and Evaluate). The 5E model has been widely accepted as the desired planning tool for science learning 
to ensure that all important components of learning are included during instruction. The 5E Model is also widely 
used as a lesson planning tool by many K-12 schools. We sought to identify parallels between desired learning 
experiences and game play experiences based on observations of the participating students’ engagement with the 
games. We felt compelled to move away from the traditional idea of content based instruction to one focused on 
why students are compelled to return again and again to a form of entertainment which requires, in some cases, more 
humiliation, more complexity than many classroom endeavors. 
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Methods 



While much of the research into the integration of games and simulations into classroom instruction has 
been based on well-established methodologies, we felt compelled to shift our paradigm by employing an adaptation 
of “experiential mode play” (EMP) (Appelman, 2007; Kolb & Fry, 1975) in order to engage in microanalysis of 
exactly what learning occurs within game play. The initial examination focused on the key strategies used by middle 
school students as they engaged in the basic training found in Call of Duty and Medal of Honor. Our main interest is 
focused solely on the instructional dimensions of gaming because as Bonk & Dennen (2004) state, due to the 
heightened popularity, video games have become integrated, in some fashion, into school curriculums because 
schools continuously explore a variety of avenues for improvement of instructional strategies that include problem 
solving, metacognition, and critical thinking skills. We clearly recognize that a video game cannot effectively be 
used as a stand-alone instructional alternative to traditional classroom routines. Additionally, we were not using the 
video games to “teach” content; rather we were intending to determine how the video games could be incorporated 
into the 5E Model. We postulate that technology has become a tool to enhance and develop a learners’ ability to 
interact cooperatively, improve analytical skills, and provide solutions to problems as they arise. This is most readily 
accomplished through careful structuring of the learning environment or curriculum interventions. 

To help organize and explain the significance of our observations, we chose to use Appelman’ s (2007) 
Experiential Mode Framework. This framework involved four phases: 

1. Pre-Play Analysis - Prior to analyzing a participant’s actions, the research team engages in 
extensive play of a particular game with careful attention paid to game structure with 
subcategories of content, environment, and affordances. Researchers create baseline 
expectations for novice and expert participation. 

2. Game Play Data Capture - Direct observation and video recording of individual player 
actions, decisions, resulting consequences, and verbal demonstrations. 

3. Game Play Data Analysis- Data is entered into an established game play analysis log used to 
observe the accomplishment of a particular goal by a player. This log is compared to the 
“ideal” game play found in the pre-play analysis. 

4. Drawing Summative Conclusions- Both qualitative and quantitative data is analyzed to 
determine player progress and performance. 

As part of the Experiential Mode Framework (Appelman, 2007), Medal of Honor and Call of Duty were 
identified as ideal game training structures because they included challenging practice, active problem solving, and 
instructional objectives. Important for younger students, the two games provided an intriguing storyline, authentic 
and reliable content, and the integration of cinematic features. Lastly, for our study, the game environments held 
clear objectives that were challenging and afforded the player a measurable goal to accomplish. We determined that 
the training sessions in both games allowed any ability level player to participate. Additionally, we chose the 
training component of the game because it included shooting objects, not people. We especially felt compelled to 
use these two games because they employed the use of “six basic categories of media used in learning such; text, 
audio, visuals, video, manipulative, and people” to move within the environment (Smaldino, et.al 2007, p. 30). 

Participants and Settings 

Participants included 6 males and 8 females recruited from a pool of thirty two 9th grade students enrolled 
at a local middle school. With the permission of parents and teachers, all of the participants were invited to engage 
in a study away from their classroom. Students were given the choice of the following games; Finding Nemo, Tony 
Hawk, Shrek, Madden06, Medal of Honor, and Call of Duty. Students who chose to play Call of Duty and Medal of 
Honor were selected as study participants. 

Data Collection 

Next, as part of the Game Play Data Capture phase of the Experiential Mode Framework, video gaming and 
post-game interviews were all conducted and video taped in the Virtual Xperience Lab (VXLAB) at a large 
Midwestern university. This facility is a standard two-room usability testing facility with an observation room 
separated from the play area with a one-way view window. The video recording captured player game movement in 
sync with a recording of the player torso. This allowed the observer to “see” exactly what game play actions cause 
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which reaction within the game. The data capture also allowed unobtrusive observations, eliminating the need for 
standing over or remaining in close proximity to players. After the students completed the training component of the 
game, interviews were conducted using a series of post game, open-ended questions. 




Data Analysis 

During our third phase of Experiential Mode Framework, Game Play Data Analysis (Figure 2) was used as 
a way to analyze player reactions. The four primary categories of player experiences are defined as; 

1. Cognition - encompassing all cognition in both cognitive and affective domains. 

2. Metacognition - encompassing all that the player is aware of, including what is perceived by 
vision, audio, olfactory, kinesthetic, and haptic senses, plus an awareness of time and any 
objects, content or information that is encountered. 

3. Choice - encompassing the player’s perception of degree of control, and access to, variables 
and information during game play. 

4. Action - encompassing the player’s perception that they can do things such as interact with 
objects and elements within the game. 



Figure 2. Game Play Analysis Log Sheet (Appelman, 2007) 



Game Play Analysis Log 



PUV6R: 
DATE; _ 



M of 



COG « I L*Aiii«d or •iijoyoil somothtivg htro 
MET = I hod to really coiiceiitiate here 
OPT * I fall lhara ware opiiom available hare 
ACT = I was able lo contiol inanipiilala things 
Off I fan lost hare 

CNT = liifoimatloii was encounlared here 
ENV « Challenges or NPR Inteiiicilons 
encountered here (explain) 

AFF * The game gave me options lo (aii^atn) 



1 TIME 
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ENV 
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From our analysis of initial observations, we found that the two key components that facilitated learning 
during the games were the presentation of clear objectives and a redundancy of instructions. For instance, several of 
our participants noted that they first listened to the oral instructions and then referred back to the written goals as a 
refresher. Four of the players had never played the game before and one did not know how to use the mouse and 
keyboard (we did not provide these instructions) to move through the game. After listening to the oral instructions 
within the game that explained which keystrokes caused which actions, the players were able to continue through 
the game. In Medal of Honor, this same student used the “ghost” icon to show where to place mock explosives. 
These observations demonstrate that, for the most part students understand requirements of video games. They were 
able to use the oral and written instructions for successful completion. 

Implications 

Conclusions to be gathered from the data is not in what type of learning players are engaged, but what 
educational researchers and the gaming industry can learn from the players. From our work with middle school 
students it is evident that a new paradigm must be considered as educators learn to teach “Millennials” (Born since 
1982, currently ages 23 and younger, Strauss, 2005). Students in this study were capable of multitasking within the 
game, even at the novice stage, and according to interviews, expected all interesting learning to include a 
redundancy of instruction/information, while still holding their attention. 




The key strategies used by players were based primarily on knowledge and experiences transferred or 
incorporated from other games they played. One player stated that he knew how to judge for trajectory of a bullet 
because he had been hunting with his father and knew that “if you miss, it’ll cost you”. When asked what the “costs” 
were he replied, “you just have to be accurate the first time or you may have to start the game all over again”. (See 
Figure 3) 

Figure 3. Shooting Target, One of the Student Milestones 



Students also noted that video games must have a compelling story line even if the media quality (picture) 
is less than optimal. All of the students, regardless of their success, asked to stay on the game longer than the 
allotted time. Several students noted that, “In the game no one is going to make fun of you.” Our work was similar 
to that found in (Squire, 2007), in the context of “gaming” mistakes and failure are simply a component of play. 
Students learn that practice and repetition within a game based learning environment is normal. Noting the intense 
engagement of all players, regardless of their experience with a particular video game, we realized that the need for 
engaged learning might also be a critical component of effective instructional settings. 
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We then incorporated our ideas into the 5E Learning Cycle (Kolb, 1984), which provides a framework for 
the design of engaging instruction. This model consists of the following phases: engagement, exploration, 
explanation, elaboration, and evaluation. The instructional emphasis for each phase is described as: 

1 . Engagement - Promotes curiosity, elicits prior knowledge, and provides an initial 
challenge that garner’ s student buy-in to the goal of the instruction 

2. Exploration - Provides a variety of opportunities for student to explore questions, 
conduct preliminary investigations, and use prior knowledge in new situations providing 
active experimentation and concrete experiences for the student. 

3. Explanation - Allows students to demonstrate their understanding of abstract concepts, 
processes or skills building and using reflective observation skills. 

4. Elaboration - Provide new experiences that extend and challenge the student 
understanding and skills. 

5. Evaluation - Encourages student to assess their understanding and abilities in relation to 
their progress toward the desired goal or outcome. 

When looking at the player experience through the framework of the 5E Learning Cycle, we see several 
aspects of the game structure that are congruent with desired learning experiences. Students expressed a need for 
clearly expressed goals and a compelling storyline (Engagement). Students quickly learned where to find 
information about how to play and reinforce their understanding of the goal. Some students may prefer to attain 
information, knowledge, and understanding through experience or affiliation with a group of experts as those found 
in the gaming community (Exploration). In several of the interviews, both female (Eigure 4) and male students 
mentioned that they were more willing to make mistakes in a game than in the classroom because their avatars 
provided a risk free setting (Evaluation). 

Figure 4. Eemale Student Engaged in Game Play 




Conclusion 

Research into simulation and games in the classroom have shown that students are more than willing to 
return to a challenging learning situation over and over if the key opportunities exist for engagement, exploration 
and evaluation. Traditional teaching methodologies that include didactic or direct transference of information may 
need to be supplemented with interactive conditions that provide these key experiences for students when they play 
video games. Using the 5E Learning Cycle, or other solid instructional design models, will ensure that all required 
components of instruction are present when using games to facilitate effective learning experiences in the classroom. 
Further research needs to be accomplished to determine specific design issues for educational games and actual 
impact on student learning. 
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